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CHAPTER  1 


INTRODUCTION 

Although  we  cannot  measure  it  precisely, 
research  and  development  activity  is  our  best 
indication  of  the  investment  in  the  scientific 
and  technological  advance  that  sooner  or  later 
results  in  productivity  growth  [36:10]. 

Productivity  growth  on  the  national  level  has  been 
defined  as  " .  .  .  the  increase  in  goods  and  services  pro¬ 
duced  per  hour  of  work  .  .  .  (lltxiii)."  The  January  1981 
Congressional  Budget  Office  report  identifies  innovation 
(technological  progress)  as  one  of  the  five  long  run  deter¬ 
minants  of  productivity  growth  (11:2-3).  Simply  put,  pro¬ 
ductivity  growth  is  the  "...  rate  of  growth  in  real  income 
per  capita  -  in  the  standard  of  living  (48:3)."  But  before 
natural  productivity  growth  can  be  determined,  productivity 
measurement  systems  must  exist  and  be  used  in  the  vast  number 
of  industries  and  government.  Man-hours,  tonnage,  volume, 
distance,  services,  facilities,  dollars,  and  various  ratios 
thereof  are  commonly  used  measures  of  inputs  and  outputs  an¬ 
nually  reported  tc  indicate  industrial,  and  ultimately 
national  productivity. 

The  major  source  of  government  productivity  measures 
is  the  Bureau  of  Labor  Statistics.  Official  productivity 
measures  are  obtained  from  separate  and  independent  surveys 
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that  are  conducted  to  determine  information  for  purposes 
other  than  for  productivity  measurement.  Measures  for  speci¬ 
fic  industries  are  published  only  when  the  data  permit  rela¬ 
tively  good  measures  (48:51-53). 

The  Department  of  Defense  (DoD)  supports  between 
one-third  and  one-fourth  of  all  the  scientists  and 
engineers  in  the  United  States ,  and  has  commensurate 
power  in  influencing  the  types  of  research  to  be  per¬ 
formed  and  the  products  to  be  developed.  Over  half 
of  the  approximately  $40  billion  spent  in  the  United 
States  each  year  on  research  and  development  comes 
from  the  federal  government,  and  of  this  national 
defense  accounts  for  more  than  half. 

This  $10  billion  or  more  spent  yearly  by  the 
Department  of  Defense  is  not  by  itself  what  makes 
defense  research  and  development  so  critically  im¬ 
portant.  Rather,  it  is  the  fact  that  the  United 
States  bases  its  overall  defense  posture  almost  en¬ 
tirely  on  "technological  superiority"  that  makes  R&D 
(Research  and  Development)  a  central  focus  for  the 
DoD  and  therefore  for  the  defense  industrial  base. 

This  large  annual  dollar  expenditure  on  R&D,  the 
concept  of  technological  superiority,  and  the  importance 
of  technological  progress  as  a  determinant  of  productivity 
growth  begs  investigation  into  the  measurement  of  government 
and  industry  R&D  productivity. 


Background 

Interest  in  the  measurement  of  R&D  laboratory  pro¬ 
ductivity  grew  from  the  investigator's  four  years  of  research 
experience  gained  with  Rome  Air  Development  Center's 
Electronic  Technology  Division  (RADC/ET) .  Exposure  to 
Research,  Development,  Testing,  and  Evaluation  (RDT&E) ,  the 
progressive  phases  of  the  R&D  process  in  DoD,  launched  this 
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inquiry  into  performance  measurement,  and  in  turn  produc¬ 
tivity  measurement  of  the  five  main  categories  of  DoD  labora¬ 
tory  responsibility  defined  in  Table  1-1  (17:INTRO-21) . 
Mansfield  defines  the  terms  "research"  and  "development"  in 
a  simpler  manner  as  follows: 

"Research"  is  original  investigation  directed 
to  the  discovery  of  new  scientific  knowledge,  and 
"development"  is  technical  activity  concerned  with 
nonroutine  problems  encountered  in  translating  re¬ 
search  findings  into  products  and  processes.  .  .  . 

Whereas  research  is  conducted  to  obtain  new  know¬ 
ledge,  development  is  required  to  reduce  the  know¬ 
ledge  to  practice.  This  often  entails  the  making 
of  various  types  of  experiments,  the  design  and  de¬ 
velopment  of  prototypes ,  and  the  construction  of 
pilot  plants  [43:6-7] . 

Similarly,  Seiler  states  "Basic  research  obviously  contri¬ 
butes  new  knowledge,  applied  research  connects  this  know¬ 
ledge  with  one  or  more  specific  products,  and  development 
prepares  these  products  for  manufacture  (58:187)."  Others 
express  essentially  the  same  ideas  (2:12;  55:3-4)  indicating 
industry  and  government  share  common  terminology. 

The  14  Air  Force  Laboratories ,  34  Army  Laboratories , 
and  22  Navy  Laboratories  listed  in  Appendix  A  are  all  involved 
in  one  or  more  of  these  five  laboratory  responsibilities  in 
the  form  of  in-house  programs,  contractual  efforts,  or  both. 
The  initial  objective  of  this  research  effort  was  to  survey 
all  Air  Force  Systems  Command  (AFSC)  laboratories  in  order 
to: 

1.  Define  the  units  of  productivity. 

2.  Identify  the  variables  affecting  productivity. 

3.  Determine  the  feasibility  of  adapting  pro¬ 
curement  productivity  indices  to  the  R&D  laboratory. 
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TABLE  1-1 

DOD  RESEARCH  AND  DEVELOPMENT  CATEGORY  DEFINITIONS 


1.  Research  -  The  initial  laboratory  purpose  is  to 
identify  and  develop  a  knowledge  base.  This  cate¬ 
gory  includes  fundamental  investigation  toward  the 
increased  understanding  of  natural  phenomena  and 
the  environment.  It  encompasses  an  attempt  to  find 
solutions  in  the  physical/  behavioral,  and  social 
sciences.  It  normally  does  not  present  a  clear 

or  direct  military  application. 

2.  Exploratory  Development  -  This  phase  embodies  efforts 
toward  the  solution  of  specific  military  problems, 
but  excludes  major  development.  Primarily,  labora¬ 
tory  experts  explore  and  develop  a  technological  base 
and  evaluate  the  practicability  and  feasibility  of 
proposed  solutions. 

3.  Advanced  Development  -  When  a  technological  base  has 
been  achieved,  projects  are  moved  into  this  phase 
using  and  creating  hardware  for  experimental  or 
operations  tests.  At  this  point,  technologies  are 
combined  to  become  a  stage  for  some  actual  produc¬ 
tion,  although  usually  tested  items  are  not  released 
for  service  or  general  use. 

4.  Engineering  Development  -  This  is  usually  the  pro¬ 
totype  stage  m  which  a  design  is  geared  for  service 
use.  Systems  emerge  at  this  level,  and  items  undergo 
extensive  performance  testing. 

5.  Management  and  Support  -  This  category  refers  to  work 
directed  toward  the  support  of  installations  and/or 
operations  required  for  general  R&D  use.  The  goal, 
from  this  perspective,  is  to  obtain  operational 
capability  through  general  support. 


4.  Measure  laboratory  performance  against  pre¬ 
determined  goals . 

5.  Measure  a  laboratory ' s  productivity  against 
itself  over  time  and  against  other  laboratories'  pro¬ 
ductivity. 

6.  Develop  a  model  to  predict  laboratory  produc¬ 
tivity. 
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Further  investigation  into  these  six  objectives  and  meeting 
with  AFSC  personnel  indicated  they  could  not  be  accomplished 
in  the  time  period  available  to  the  investigator,  but  they 
remain  as  potential  long-term  objectives  for  future  studies. 
The  objectives  of  this  research  effort  then  evolved  to  the 
following: 


1.  Conduct  an  extensive  literature  review  of 
public  and  private  publications  to  determine  the 
evolution  of  R&D  productivity  measurement  from  1960 
to  the  present  (state-of-the-art) . 

2.  Determine  the  qualitative  and  quantitative 
measures  of  productivity,  theoretical  and  empirical. 

3.  Identify  the  prevalent  variables  affecting 
productivity . 

4.  Identify  literature  similarities,  contra¬ 
dictions,  and  novelties. 

5.  Interview  Industry  and  DoD  laboratories  to 
determine/identify  currently  used  productivity 
measurement  techniques  and  variables,  if  any. 


Definitions 

This  investigation  into  the  measurement  of  produc¬ 
tivity  in  R&D  laboratories  requires  that  a  consistently  de¬ 
fined  and  applied  set  of  terms  be  included  for  clear  compre¬ 
hension.  The  terms  are  defined  in  Table  1-2  and  retain  their 
definitions  throughout  this  thesis.  The  terms  productivity, 
efficiency,  and  effectiveness  were  also  verbally  defined 
during  each  laboratory  telephone  interview  to  maintain  com¬ 
monality  in  word  meaning  between  each  laboratory. 
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TABLE  1-2 


PRODUCTIVITY  DEFINITIONS 


X.  PRODUCTIVITY:  "The  productivity  of  an  organization  may 
b*  broadly  definad  as  tha  afficiancy  with  which  its 
resources  are  utilized  to  produce  final  outputs  (53:4).* 
Productivity  is  a  product  of  efficiency  and  effective¬ 
ness.  In  equation  form 

PRODUCTIVITY  -  PERFJRMANCE/RESOURCES 
-  DIRECTED  OUTPUT/INPUT 

PRODUCTIVITY  IMPROVEMENT  -  INCREASE  OUTPUT/GIVEN 

INPUTS 

or 

■  GIVEN  OUTPUT/DECREASED 
INPUTS 

or 

-  INCREASED  OUTPUT/ 
DECREASED  INPUTS 

2.  EFFECTIVENESS:  *Ef factiveness  concerns  the  extent  to 
which  government  programs  achieve  thair  objectives 
(53:5).*  Effectiveness  identifies  the  degree  to  which 
the  laboratory  accomplishes  its  mission. 

3.  EFFICIENCY:  Efficiency  refers  to  the  ratio  of  output 
to  input  and  implies  ’more,  better,  faster  and 
cheaper  .  .  -  (1:13)."  Efficiency  concerns  the  utili¬ 
zation  of  resources  to  execute  laboratory  programs  and 
functions  at  minimal  cost  (53:5). 

4.  OBJECTIVE  MEASURES:  Objective  measures  are  quanti¬ 
fiable  resources  and  performance  measures.  These  mea¬ 
sures  may  be  direct  or  indirect. 

5.  SUBJECTIVE  MEASURES:  Subjective  measures  are  qualita¬ 
tive  determinants  of  resource  utilization  and  perfor¬ 
mance. 

6.  INVENTION:  Invention  is  "the  creation  of  an  idea  and 
its  reduction  to  practice;  the  first  stage  of  innova¬ 
tion  (2:8).*  Invention  "aims  at  the  practical  appli¬ 
cation  of  a  principal  which  may  be  fully  or  only  par¬ 
tially  apprehended  (55:4)." 

7.  INNOVATION:  Innovation  is  "the  bringing  of  an  inven¬ 
tion  into  widespread,  practical  use  (2:8),*  and  the 
"change  with  respect  to  existing  practices  (70:35).* 

8.  CREATIVITY:  Creativity  is  ’the  process  in  which  either 
original  or  imaginative  ideas  are  conceived  or  communi¬ 
cated  (47:873)." 


To  properly  determine  productivity  in  an  R&D 
laboratory,  productivity  indicators  must  either  directly 
or  indirectly,  and  objectively  or  subjectively,  measure 
laboratory  inputs  (budget,  man-hours,  mental  activity.  .  .) 
and  outputs  (papers,  contracts,  briefings,  patents.  .  .). 
Objective  information  is  more  desirable  than  subjective 
information  because  quantified  information  lends  itself 
to  comparison  and  reduces  prejudicial  distortion.  The 
phrase  "directed  outputs"  (performance)  is  preferred  over 
the  term  outputs  because  it  specifically  indicates  perfor¬ 
mance  directed  toward  an  R&D  objective  rather  than  just  the 
accomplishment  of  some  task  or  service.  Good  measures 
should  incorporate  the  concepts  of  reliability,  consistency, 
stability,  and  validity,  as  identified  by  Szilagyi  and 
Wallace  (67:447-453),  because  these  concepts  "refer  to  the 
adequacy  of  the  information  that  is  generated  and  employed 
in  subsequent  decisions.  .  .  (67:449)."  The  reference  per¬ 
tains  to  individual  performance  measurement,  but  is  directly 
applicable  to  the  performance  measurement  of  an  R&D  organi¬ 
zation. 

The  terms  inventions,  innovation,  and  creativity  are 
frequently  associated  with  scientific  output,  and  are  there¬ 
fore  included  in  Table  1-2.  The  definitions  indicate  the 
difficulty  of  formulating  indicators  to  measure  these  types 
of  outputs.  Invention  disclosures  and  patents  may  indicate 
the  quantity  of  output,  but  a  subjective  determination  of  the 


7 


quality  of  the  invention  disclosures  and  patents  should 
also  be  made,  which  does  not  easily  lend  itself  to  compari¬ 
son  with  other  invention  disclosures  and  patents.  Wide¬ 
spread  implementation  of  a  medical  diagnostic  technique 
is  an  example  of  an  innovation,  but  time  is  the  critical 
factor.  How  widespread  is  widespread,  and  how  much  change 
is  required?  Either  response  is  still  most  likely  several 
years  removed  from  the  initial  invention.  The  communica¬ 
tion  of  original  ideas  is  easier  to  measure  than  the  pro¬ 
cess  of  idea  conception,  but  the  subjectivity  problem  re¬ 
curs.  These  examples  clearly  indicate  the  types  of 
problems  which  must  be  dealt  with  in  measuring  R&D  pro¬ 
ductivity. 

Statement  of  the  Problem 

This  seeming  dilemma  has  given  rise  to  the  need  to 
identify  qualitative  and  quantitative  factors  measuring  pro¬ 
ductivity  in  public  and  private  R&D  laboratories.  An  effec¬ 
tive  measure  of  productivity  has  not  been  identified  because 
common  measures  for  laboratory  productivity  have  not  been 
defined.  This  problem  exists  because  of  the  task  characteris¬ 
tics  unique  to  the  laboratory  research  and  development  func¬ 
tion;  i.e.,  protracted  work  cycles  and  the  absence  of  an 
identifiable  product. 

Scope  and  Limitations 

This  study  examines  the  theoretical  and  empirical 


publications  concerning  the  collective  productivity 


measurement  of  scientists ,  engineers ,  and  their  managers , 
and  determines  current  laboratory  perceived  productivity 
indicators  used  to  measure  productivity  in  DoD  and  selected 
industry  laboratories.  The  types  of  R&D  laboratory  person¬ 
nel  investigated  include  chemists,  physicists,  engineers, 
technicians,  metallurgists,  logisticians,  work  effort 
leaders,  and  branch  chiefs.  Specifically  not  included  are 
administrative  (secretarial)  employees  and  high  level  labora¬ 
tory  managers . 

This  thesis  is  limited  to  the  study  of  R&D  produc¬ 
tivity  measurement.  Productivity  improvement,  productivity 
enhancement,  and  factors  affecting  laboratory  productivity 
are  not  explored,  but  an  expanded  Related  Sources,  contained 
in  the  Bibliography  section,  is  included  to  serve  as  an  ini¬ 
tial  step  for  follow-on  study  of  these  related  topics. 
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CHAPTER  2 


RESEARCH  METHODOLOGY 

This  investigation  into  the  state-of-the-art  of 
productivy  measurement  in  the  research  and  development 
community  consisted  of  three  distinct  areas  of  interest 
identified  as  follows: 

1.  A  review  of  the  empirical  literature  of 
industry  and  DoD  laboratories. 

2.  A  review  of  the  theoretical  literature  of 
industry  and  DoD  laboratories. 

3.  An  extensive  telephone  interview  of  industry 
and  DoD  laboratories. 

These  three  interests  were  pursued  using  two  methodologies: 
an  exhaustive  search  of  the  theoretical  and  empirical  litera¬ 
ture,  and  an  extensive  telephone  interview  process. 

Literature  Resources 

The  empirical  and  theoretical  literature  reviews 
were  conducted  simultaneously  using  the  following  four  ser¬ 
vices  : 

1.  Defense  Technical  Information  Center  (DTIC) ; 

2.  Lockheed  Data  Log; 

3.  National  Technical  Information  Service  (NTIS) ; 

4.  National  Referral  Center  (NRC) . 

These  library  and  national  services  provided  the  majority 
of  all  references  used.  These  are  limited  however  for  two 
reasons:  first,  they  are  totally  dependent  on  searchers 

using  the  proper  key  words,  and  secondly,  all  useful 
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publications  may  or  may  not  be  entered  into  the  Systems. 

Text  book  references  were  found  from  the  card  catalogues  at 
the  Wright  State  University  Library,  the  Air  Force  Institute 
of  Technology  Libraries,  and  the  Wright-Patterson  APB 
Technical  Library.  The  key  words  used  in  the  searches  are 
listed  in  Table  2-1.  The  term  evaluation  is  not  included 
in  the  listing  because  nearly  every  article  located  with 
this  key  word  referred  to  evaluating  alternative  research 
projects  prior  to  laboratory  investigation,  and  did  not 
evaluate  completed  or  on-going  research  projects. 

Additional  literature  references  and  information 
were  also  obtained  from  personal  telephone  interviews  (con¬ 
ducted  by  the  author)  with  recognized  experts  in  the  field 
of  productivity  measurement.  Each  publication  was  classi¬ 
fied  as  theoretical  or  empirical  and  was  reviewed  accordingly. 
Empirical  literature  was  characterized  by  a  specific  sample 
or  population,  whereas  theoretical  publications  were  not. 

Many  publications  contain  a  combination  of  significant  theo¬ 
retical  and  empirical  information.  R&D  productivity  measure¬ 
ments  are  more  of  an  indirect  than  direct  nature-  In  an  ef¬ 
fort  to  specifically  identify  the  most  commonly  used  indi¬ 
cators,  specific  partial  productivity  indicators  (output/ 
partial  input)  are  pursued  rather  than  a  Total  Factor 
Productivity  (TFP)  measurement  (42:40),  which  is  essentially 
a  composite  of  all  partial  productivity  measurements  in  com¬ 
mon  terms. 
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TABLE  2-1 
KEY  SEARCH  WORDS 


DTIC ,  NTIS,  LOCKHEED  DATA  LOG,  NRC 

CARD  CATALOGUE 

Air  Force  Research 

Effectiveness 

Army  Research 

Efficiency 

Army  Laboratories 

Empirical 

Industrial  Research 

Innovation 

Military  Research 

Measures 

Naval  Research 

Performance 

Naval  Research 

Laboratories 

Productivity 

Research 

Research  and 

Productivity 

Productivity  Variables 

Reliability 

Research 

Development 

Research  Facilities 

Research  Laboratories 

Research  and 

Development 

Theoretical 

Variables 

Validity 

Telephone  Interview 

In  addition  to,  and  to  complement  the  literature 
review,  original  data  was  collected  by  the  author  through 
an  extensive  telephone  interview  effort  designed  to 
identify  "state-of-the-art"  productivity  measurement  cri¬ 
teria  (organizational  and  individual)  as  used  by  both  DoD 
and  selected  private  industry  laboratories.  The  DoD 
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Laboratories  interviewed  consisted  of  those  laboratories 


* 


identified  in  the  publication  Department  of  Defense  In-House 
RDT&E  Activities,  30  October  1979  ( 6 9 : xxv-xxx) .  DoD 
laboratory  telephone  numbers  were  obtained  from  the  publi¬ 
cation  Institutional  Barriers  on  DOD  Laboratories:  A  Report 
of  the  Ad  Hoc  Task  Group  on  In-House  Laboratories  to  the 
Deputy  Under  Secretary  for  Research  and  Advanced  Technology 
(52:14-29).  The  industry  laboratories  were  selected  from  the 
list  of  participants  who  attended  the  April  21-24,  1980 
International  Conference  on  Productivity  Research  in  Houston, 
Texas . 

The  specific  questions  are  provided  in  Appendix  B. 

The  tabled  results  and  discussion  are  included  in  Chapter  4. 
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CHAPTER  3 


CHRONOLOGICAL  LITERATURE  REVIEW 


Background 

An  exhaustive  literature  search  of  the  theoretical 
and  empirical  publications  concerning  R&O  productivity  re¬ 
sulted  in  the  13  theoretical  publications  and  41  empirical 
publications  chronologically  compiled  in  Appendix  C.  Em¬ 
pirical  publications  are  distinguished  from  theoretical  pub¬ 
lications  by  the  fact  that  they  are  characterized  by  the 
study  of  the  behavior  of  a  sample  or  population. 

The  theoretical  table  (Table  C-l )  includes  the  date, 
author.fs)  ,  title  of  the  publication,  data  collection  metho¬ 
dology,  the  degree  of  statistical  analysis  (if  any) ,  the 
primary  type  of  productivity  measurement  (subjective,  ob¬ 
jective,  or  both),  and  the  primary  thrusts  of  the  articles. 
The  empirical  table  (Table  C-2)  also  includes  the  date, 
author(s),  title  of  the  publication,  data  collection  method¬ 
ology,  type  of  productivity  measurement  (subjective,  objec¬ 
tive,  or  both),  degree  of  statistical  analysis,  subject 
category,  and  sample  or  population  size. 

Table  3-1  is  the  key  to  the  tables  in  Appendix  C. 

The  data  collection  methodology  column  is  self-explanatory. 
The  types  of  productivity  measures  used  (subjective  or  ob¬ 
jective)  were  determined  based  on  the  definitions  given  in 
Table  1-2.  The  degree  of  statistical  analysis  colum  refers 
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TABLE  3-1 


KEY  TO  LITERATURE  REVIEW  TABLES  (APPENDIX  C) 


DATA  COLLECTION 
METHODOLOGY 

TYPE  OF 
PRODUCTIVITY 
MEASURE 

DEGREE  OF 
STATISTICAL 
ANALYSIS 

SUBJECT  CATEGORY 

Su*Survey  or 
Questionnaire 

0»0b jective 

Co=Complete 

G= Government 

In=Interview 

S«Sub jective 

M-Model 

I^Industry 

C*Cases 

Nondirective 

Observer 

N=None 

U=»University 

F=*Field  Study, 
Variable 
Manipula¬ 
tion  , 

Experimen¬ 

tal 

P*Percen- 
tages 
or  ratios 

LR=Literature 

Review 

SD=Standard 

Devia¬ 

tion 

to  the  highest  level  of  analysis  from  none  (0)  to  Complete 

(correlations).  The  Subject  Category  column  included  respon¬ 
dents  from  government,  industry,  or  both  government  and  in¬ 
dustry  laboratories  (personnel) .  The  remainder  of  this 
chapter  is  devoted  to  summarizing  the  literature  review 
findings. 


Theoretical  Review 


The  survival  of  a  business  depends  on  per¬ 
formance  in  eight  "key  result  areas":  marketing; 
innovations;  human  organizations;  financial 
resources;  physical  resources;  productivity; 
social  responsibility;  and  profit  requirements. 
Drucker  argues  that  the  specific  goals  in  any 
one  of  these  key  result  areas  will  vary  from 
one  business  firm  to  another  [1:56]. 
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The  major  barrier  to  understanding  the  relationship 
between  organizational  setting  and  scientific  develooment 
"lies  in  the  difficulty  of  developing  techniques  to  quanti¬ 
tatively  and  qualitatively  measure  scientific  output 
(26:192)."  Gordon  (26:192)  stated  in  1963: 

.  .  .  that  the  use  of  summaries,  in  conjunction 
with  the  "panel"  technique  in  which  eminent  authori¬ 
ties  in  given  fields  are  used  to  evaluate  scienti¬ 
fic  accomplishment,  may  provide  a  practical  solution 
to  the  problem  of  qualitatively  assessing  recent 
research  accomplishment. 

The  proposed  panel  technique  was  used  to  evaluate  medical 
sociology  studies  on  a  ten-point  scale  in  each  of  the 
following  four  criteria: 

1.  Research  problem  importance; 

2.  Research  productiveness  -  the  extent 
to  which  the  research  adds  to  the  existing 
knowledge  base  from  established  lines  of 
research;  extensions  of  previous  theory ; 

3.  Research  innovations  -  the  extent 
to  which  the  research  adds  to  the  existing 
knowledge  base  through  new  lines  of  research, 
development  of  new  theoretical  statements; 

4.  Overall  significance  of  the  findings. 

The  results  indicate  that  the  technique  may  provide  a 
solution  to  the  problem  of  qualitative  assessment,  but 
that  the  technique  may  not  apply  to  all  areas  of  research. 

Villers  (71:10)  suggested  the  following  three  ground 
rules  in  1964  regarding  the  estimates  made  by  research 
people  of  the  time  and  money  needed  to  reach  a  specific 
technical  goal: 

1.  A  reasonable  range  of  variance  should  be 
agreed  upon  when  the  estimate  is  submitted  to 
eliminate  the  temptations  to  provide  for  a  cush¬ 
ion.  In  may  cases,  the  estimator  may  be  tempted 
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to  delay  final  completion  to  avoid  the  charge  of 
having  provided  a  cushion. 

2.  Absolute  accuracy  should  not  be  expected. 

As  long  as  the  research  people  are  within  the 
anticipated  range  of  variance,  their  performance 
should  be  considered  as  successful. 

3.  As  soon  as  the  research  people  discover 
that  they  are  likely  to  exceed  the  expected  range 
of  variance,  they  should  report  the  need  for  more 
time  and  money.  Management  should  expect  such 
requests  and  accept  them  without  automatically 
objecting  but  should  look  into  too  frequent  re¬ 
quests,  which  obviously  indicate  abuses. 

Broadwell  (6:150)  states  that: 

Engineering  Productivity  is  a  function  of 
satisfaction,  both  desired  and  obtained,  and 
is  proportional  to  ability. 

.  .  .  The  engineer's  work  varies  greatly 
from  the  normal  measurable  productivity-type 
jobs,  however,  since  much  of  an  engineer's 
value  lies  in  preventing  work — work  that  is 
not  necessary  or  deferrable.  Thought  and 
computations — both  necessary  and  time  con¬ 
suming — by  the  engineer  can  often  prove  that 
there  is  a  better,  and  perhaps  simpler,  way 
of  doing  a  job,  or  as  has  been  said,  the  job 
may  be  proven  unnecessary  altogether.  In 
such  cases ,  little  or  no  production-type 
work  is  visible.  Certainly  no  measurable 
work  has  been  performed. 

So  we  are  speaking  of  a  different  mean¬ 
ing  when  we  refer  to  Engineering  Productivity. 

We  cannot  measure  it  by  any  given  standard. 

A  series  of  graphs  are  developed  to  emphasize  that 
satisfaction  and  ability  affect  output  to  a  significant 
degree.  Engineering  output  depends  on  the  engineer's  abil¬ 
ity  (the  right  man  in  the  right  job)  and  the  trend  of  the 
engineer's  satisfaction  (is  the  engineer  more  or  less  satis¬ 
fied  now  than  last  quarter  in  terms  of  money,  time,  inter¬ 
rests,  and  other?).  Broadwell  states  that,  "if  we  skillfully 
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handle  the  management  responsibilities  given  to  us,  we  can 
provide  a  healthy  and  productive  atmosphere  within  the  en¬ 
gineering  organization  (6:153}." 

In  order  to  refine  the  meaning  of  scientific  research 
efficiency,  and  to  identify  some  specific  methods  to  ob¬ 
jectively  measure  it,  Lipetz  wrote  a  book  in  1965  based 
on  a  doctoral  thesis  submitted  more  than  six  years  earlier. 

( 40 : x-xi) . 

In  scientific  research,  however,  it  is  difficult 
to  state  with  assurance  just  what  the  product  should 
be;  hence,  it  is  difficult  to  know  what  it  is  in  the 
performance  of  research  organizations  that  the  admin¬ 
istrator  should  try  to  measure  £40:22]. 

.  .  .  G.  A.  W.  Bolhm  asserts,  "Without  doubt  .  .  . 
research  is  the  most  difficult  of  all  business  opera¬ 
tions  to  measure  and  control."  The  apparent  non- 
repetitive  nature  of  the  objectives  of  research 
organizations  suggests  to  many  administrators  that 
a  measurable  unit  of  scientific  research  productivity, 
if  developed,  could  be  valid  only  within  a  single 
organization;  and,  even  there,  it  could  retain  its 
validity  for  only  a  short  time  [40:22], 

The  investigation  of  the  conclusion  that  research  organiza¬ 
tions  and  research  administrative  techniques  can  not  be 
objectively  compared  is  the  major  objective  of  Lipetz' 
work. 


A  useful  measurement  must  satisfy  three  criteria: 

1.  The  measurement  must  be  pertinent.  The 
measurement  must  relate  to  a  problem  which  is  im¬ 
portant  to  the  administrator. 

2.  The  measurement  must  relate  to  a  problem 
which  is  recurrent  in  administration  and  which 
makes  the  comparison  of  different  measurement  re¬ 
sults  meaningful. 

3.  The  measurement  must  be  practical.  The 
processes  of  making  or  interpreting  the  measurement 
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must  not  introduce  adverse  affects  on  the  organiza¬ 
tion  which  will  outweigh  the  improvement  to  be 
gained  from  the  use  of  the  measurement  [40:31]. 

Lipetz  reports  the  results  of  several  surveys  taken 
in  both  industrial  research  organizations  by  Anthony  and 
Day,  and  in  government-sponsored  research  organizations  in 
the  United  States  and  Great  Britain  by  Hiscocks.  The  research 
by  these  authors  indicate  "...  that  no  satisfactory  means 
for  evaluating  the  effectivness  of  research  are  now  in 
use  (40:33),"  as  of  1965. 

A  majority  of  the  measures  used  at  that  time  to 
determine  research  performance  were  of  the  following  five 
types : 


1.  Measurement  of  personnel.  Counts  of  the 
number  of  personnel  on  the  staff  with  different  de¬ 
grees  of  education  or  responsibility,  counts  of 
staff-hours  consumed  for  different  degrees  of  edu¬ 
cation  or  responsibility. 

2.  Measurement  of  space.  Determinations  of 
the  amount  of  building  volume  or  floor  space  given 
over  to  different  activities  or  different  types 

of  personnel  or  equipment. 

3.  Measurement  of  capital  assets.  Determina¬ 
tions  of  the  value  of  buildings  and  equipment  de¬ 
voted  to  different  activities  or  to  different  types 
of  personnel  or  service  support  for  different  ac¬ 
tivities  or  different  types  of  personnel. 

4 .  Measurement  of  expenses .  Determinations  of 
the  cost  of  salaries,  supplies,  or  service  support 
for  different  activities  or  different  types  of 
personnel. 

5.  Ratios.  Calculations  of  the  ratio  of  any 
of  the  above  measurements  to  any  other,  e.g.,  total 
annual  expense  per  "senior  researcher",  capital 
investment  per  staff  member,  technicians  per  scien¬ 
tist,  floor  space  per  employee  [40:34]. 

While  these  measures  meet  the  criteria  of  a  useful  measure¬ 
ment,  they  are  insufficient  as  they  do  not  indicate 
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the  value  of  work  performed,  the  quantity  of  work  performed, 
or  measure  the  effectiveness  of  a  research  organization. 

For  the  research  administrator  the  achievements 
of  the  research  organization  are  to  be  found  not  in 
repetitious  activities ,  but  in  activities  which  create 
things  that  are  new  and  of  scientific  value  [40:371. 

Some  organizations  (40:38)  use  the  ratio  of  sales 
of  new  research-developed  products  to  sales  of  traditional 
products  to  measure  the  effectiveness  of  research.  But  this 
ratio  does  not  consider  costs  as  well  as  achievements,  indi¬ 
cating  the  ratio  of  profits  of  new  products  to  traditional 
products  may  be  appropriate. 

"The  really  serious  objection  to  the  use  of  sales 
figures  in  determining  the  effectiveness  of  research 
is  that  they  tell  us  too  little  about  the  actual  re¬ 
search  group — how  well  it  functioned,  how  much  better 
it  could  have  done  judging  from  other  research  groups. 
Sales  figures  are  influenced  by  too  many  factors 
which  are  unrelated  to  the  efforts  and  achievements 
of  the  research  group  [40:38]. 

Another  method,  reportedly  used  in  the  USSR 
(40:39-40),  is  based  upon  the  percentage  of  achievement  of 
research  objectives.  In  this  method,  a  laboratory  submits 
a  list  of  research  objectives  to  a  reviewing  body  for  ap¬ 
proval  and  finding.  The  approved  list  is  compared  with  ac¬ 
tual  research  objectives  accomplished  at  the  end  of  the  fis¬ 
cal  year,  and  the  "percentage  fulfillment"  is  determined. 
The  method  is  improved  when  the  number  of  objectives  accom¬ 
plished  per  unit  cost  (money,  time,  manpower,  etc.)  is  con¬ 
sidered,  and  when  a  weighting  is  placed  on  the  individual 

objectives  in  an  attempt  to  create  an  appropriate  value 
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for  each  objective  since  each  objective  cannot  be  of  equal 
value . 


The  obvious  flow  in  measurements  of  publications 
produced  as  indicators  of  research  effectiveness  is 
the  implicit  assumption  that  all  research  publica¬ 
tions  are  of  equal  value,  an  assumption  which  is 
quite  false,  as  all  those  who  are  acquainted  with 
scientific  research  are  well  aware.  .  .  .  Nevertheless, 
some  confidence  may  be  placed  in  measurements  of 
this  type  when  there  are  large  numbers  of  measure¬ 
ments  involved  and  their  interpretation  is  statis¬ 
tical  [40:42] . 

(Lipetz  uses  the  terms  efficiency  and  effectiveness  synomy- 
nously.)  Research  shows  that  "statistically,  there  is  a 
strong  correlation  between  a  high  rate  of  publication  and 
a  scientist's  receipt  of  high  academic  honors  (40:42)." 

The  production  of  publications  is  a  type  of  re¬ 
search  achievement,  and  is  a  practice  common  to  most 
research  organizations.  In  many  research  organiza¬ 
tions,  publications  constitute  the  only  tangible  ac- 
chievements.  .  .  .  The  making  of  such  measurement 
would  not,  in  most  research  organizations,  be  ap¬ 
preciably  costly  or  disruptive.  Because  this  type 
of  measurement  is  so  attractive,  it  is  appropriate 
to  investigate  it  further  to  see  whether  some  way 
can  be  found  to  weight  or  modify  the  units  counted 
so  that  the  measurements  produced  will  be  consistent 
with  the  relative  values  of  the  publications  to 
which  they  pertain  [40:42-43], 

Lipetz  suggested  that  the  following  guidelines 
should  be  used  to  measure  R&D  productivity  (40:93) : 

1.  Quantifiable  objectives  must  be  estab¬ 
lished. 

2.  Follow  the  scientific  method. 

a)  familiarity  with  existing  knowledge; 

b)  hypothesis  created,  based  on  know¬ 
ledge  and  relationships  in  a) ; 

c)  prediction  of  new  knowledge  deduced; 

d)  test  validity  of  the  prediction; 

e)  new  knowledge  results  from  testing; 

f)  cycle  to  a) . 
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Additionally,  Lipetz  identifies  six  types  of  desirable  and 
measurable  output  of  scientific  research  activity: 

1.  Description; 

2.  Definition; 

3.  Hypothesis; 

4.  Explanation; 

5.  Prediction; 

6.  Experimental  technique. 

Each  of  these  elements  seems  to  be  identified  in  quan 
titative  units  under  certain  circumstances .  .  .  .  These 
measurements  would  make  possible  the  computation  and  ex¬ 
change  of  figures  on  the  efficiency  of  scientific  achieve 
ment  in  scientific  research  organizations  [40:94]. 

Lipetz  (40:94)  states  that  each  of  the  above  six  outputs  can 
be  subdivided  to  include  nearly  every  other  element  of  iden¬ 
tifiable  and  measurable  scientific  output. 

Lipetz  continues  with: 

.  .  .  the  reason  for  trying  to  develop  new 
measurements  of  research  production  is  to  make  it 
easier  for  administrators  to  profit  from  each  other's 
experience  by  facilitating  the  interchange  of  in¬ 
formation  which  correlates  administrative  methods 
employed  with  efficiences  obtained.  Quantitative 
information  is  desirable  because  it  carries  a  more 
definite  meaning  than  qualitative  information.  How¬ 
ever,  the  reliability  of  the  information  which  an  ad¬ 
ministrator  receives,  whether  quantitative  or  quali¬ 
tative,  must  be  determined  to  his  satisfaction  so 
that  he  may  use  the  information  wisely.  ...  If  the 
information  in  question  is  a  subjective  evaluation, 
there  is  no  way  to  verify  its  reliability  except  to 
make  a  subjective  evaluation  of  the  evaluator.  If 
the  information  in  question  is  an  objective  evalua¬ 
tion,  based  upon  something  accessible  and  enduring, 
its  reliability  may  be  confirmed  with  much  greater 
precision  and  without  reference  to  the  original 
evaluator  [40:109]. 

The  R&D  evaluation  and  measurement  process  may  con¬ 
sist  of  objective  (quantifiable) ,  subjective  (nonquanti- 
fiable) ,  or  a  combination  of  both  categories  of  productivity 
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measurement  criteria.  Roman  (56:390)  stated  the  following: 

Evaluation  involves  measurement  and  should  be 
related  to  pertinent  objectives  and  standards. 

Determining  objectives  is  possible  in  research 
and  development,  but  establishing  standards  is 
extremely  difficult,  as  we  have  seen,  because  the 
work  is  usually  nonrepetitive  and  lacks  prece¬ 
dent. 

Evaluation  and  measurement  of  R&D  effort  should 
start  with  an  appraisal  of  the  individual.  It 
may  subsequently  survey  group  performance,  which 
can  be  greater  or  less  than  the  sum  of  the  contri¬ 
butions  of  individual  members.  The  organization's 
achievement  must  also  be  evaluated  as  a  composite 
of  individual  and  group  efforts.  Product  evalua¬ 
tion,  which  is  more  tangible,  provides  a  separate 
yardstick  against  which  to  check  the  more  intan¬ 
gible  assessment  of  human  activity  [56:390], 

Roman  further  pursued  the  thought  that  the  individual 
in  the  R&D  organization  should  be  the  starting  point  for 
R&D  evaluation  and  measurement  as  follows: 

The  productive  unit  in  R&D  organizations  revolves 
around  the  individual.  Although  he  is  only  part  of 
the  total  evaluation  and  measurement  problem,  it  is 
necessary  to  understand  the  individual's  performance 
in  order  to  attempt  a  more  comprehensive  input-output 
appraisal  of  R&D  effort. 

Creative  performance  is  highly  unpredictable  and 
impossible  to  quantify  validly  with  existing  methods. 
Individuals  doing  creative  work  have  different  abili¬ 
ties,  and  there  are  no  universally  accepted  standards 
of  efficiency  in  identifying  and  solving  problems 
[56:390-911. 

In  1968,  Roman  outlined  the  following  as  reasons  for 
needing  R&D  performance  measurements : 

1.  R&D  expenditures  are  increasing; 

2.  Scarcity  of  R&D  specialists; 

3.  Comparatively  high  salaries  of  R&D 
specialists ; 

4.  Rising  operating  costs; 

5.  Competitive  pressure  for  technological 
progress  and  stable  price  levels; 

6.  A  considerable  portion  of  R&D  expendi¬ 
ture  continues  to  be  concentrated  on  the  defense 
effort; 
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7.  Rising  weapons  system  costs  forcing  project 
tradeoffs; 

8.  A  need  to  measure  individual  mental  perfor¬ 
mance  [56:392-93]. 

Roman  further  oelieved  appraisal  methods  should  include 
both  job-related  activity  and  personality  factors  in 
"proper  balance",  yet  distinguish  between  the  two. 

Some  of  the  more  applicable  factors  to  consider 
in  an  appraisal  are  technical  competence,  potential 
technical  growth,  managerial  promise,  productivity, 
quality  of  work,  promotion  potential,  creativity, 
ability  to  work  with  people,  personal  habits  and 
appearance,  and  cost  consciousness  [56:393]. 

Roman  stated  that  there  is  a  lack  of  quantifiable 
factors  in  the  several  individual  appraisal  techniques 
used  in  R&D  organizations  as  of  1968.  These  techniques  are 
as  follows : 


1.  The  forced  comparison  technique  (yields 
relative  position) ; 

2.  The  totem  pole  (correlates  relative  position 
with  salary) ; 

3.  The  check  sheet  (evaluates  individual 
characterizations) ; 

4.  The  McGreggor  method  (subordinate  cites 
his/her  own  goals,  and  self  evaluates  his/her  pro¬ 
gress)  ; 

5.  The  publications  standard  (number  of  arti¬ 
cles  published  in  technical  journals)  [56:393-395], 

Roman  provided  a  short  discussion  describing  the  faults  with 

each  technique  and  the  possible  problems  of  rater  bias,  rater 

consistency,  validity  of  appraisals,  and  personality  conflicts. 

Over  the  long  run,  for  lack  of  more  acceptable 
standards,  profitability  is  used  as  the  most  con¬ 
crete  measure  of  an  organization's  performance.  .  .  . 

In  government,  military,  and  nonprofit  units,  pro¬ 
fit  is  not  a  consideration,  and  the  vagueness  of 
the  standards  used  to  evaluate  the  organization 
makes  assessments  of  individual  effort  relatively 
indeterminate  [56:396]. 
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In  1968,  Roman  (56:397-703)  proposed  identifying 
project  components  and  establishing  a  work  management  system 
to  relate  individual  assignments  to  project  objectives.  The 
system  basically  consists  of  periodically  charting  the  pro¬ 
gress,  budget,  time  and  money  expenditures  on  a  standardized 
project  support  assignment  form,  and  for  each  individual  and 
project  on  a  work  standardized  form.  This  form  indicates 
the  individual's  responsibilities,  time  schedule,  and  pro¬ 
gress.  The  Weekly  Backlog  method  is  a  simpler  alternative 
measuring  instrument  which  records: 

1.  current  work  in  progress; 

2.  new  work  initiated; 

3.  work  completed. 

The  evaluation  of  a  group,  which  may  represent  a 
function,  project,  or  technical  subdivision,  may  be  deter¬ 
mined  as  the  collective  assessment  of  the  individual's  per¬ 
formance;  1)  how  well  the  project's  objectives  are  accom¬ 
plished,  2)  the  schedule,  number  of  projects  accepted,  in 
process,  and  completed,  and  3)  expenditures  versus  achieve¬ 
ments  (56:404-405). 

Roman  contended  that  the  two  best  standards  for  mea¬ 
suring  organizational  performance  are  relative  growth  and 
profitability,  but  these  standards  cannot  be  applied  to 
government,  military,  and  nonprofit  organizations.  There¬ 
fore,  new  methods  need  to  be  developed  to  measure  organiza¬ 
tional  performance.  The  concepts  of  effectiveness  (ability 

to  achieve  its  objectives)  and  efficiency  (degree  to  which 
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objectives  actually  achieved)  are  introduced  and  defined  as 
possible  measures  of  organizational  performance.  The  order 
of  flow  thus  becomes  1)  established  organizational  objectives, 
2)  organizational  operation,  3)  analyze  organizational  ef¬ 
fectiveness,  and  4)  analyze  organizational  efficiency. 

"A  long  range  program  will  give  management  some 
indication  of  how  total  R&D  is  contributing  to  overall  or¬ 
ganizational  objectives  (55:408). "  Roman  suggested  that  an 
organization  interrogate  itself  with  the  following  questions 
to  help  determine  R&D  contributions: 

1.  Is  long-range  R&D  planning  related  to  over¬ 
all  organizational  goals? 

a.  Is  the  plan  formalized  and  maintained 
on  a  continuing  basis? 

b.  Does  the  long  range  plan  identify  basic 
research,  applied  research,  and  development 
within  the  existing  organization  structure  and 
in  relation  to  the  existing  functional  activi¬ 
ties? 

d.  Do  the  policy-making  managers  in  the 
research  group  understand  the  long-range  plan 
and  its  connection  with  corporate  objectives? 

2.  Are  projects  consistent  with  the  long  range 
R&D  plan,  and  are  they  being  reviewed  and  updated  on 
a  continuous  basis? 

3.  How  much  of  the  organization's  earnings  are 
derived  from  products  which  have  been  developed  by 
research  in  the  last  5,  10,  or  15  years? 

4.  How  many  of  the  products  and  product  im¬ 
provements  were  introduced  by  the  organization 
rather  than  by  competitors? 

5.  What  is  the  relationship  between  the  use 
and  quality  standards  of  the  products  being  turned 
out? 

6.  How  many  new  products  are  based  on  in-house 
research  in  contrast  with  licensing  arrangements 
and  patent  purchases? 

7.  What  is  the  share  of  the  market? 

8 .  How  do  patents  granted  compare  with  those 
granted  to  competitors  in  number  and  scope? 

9.  Has  research  been  a  factor  in  cost  reduc¬ 
tion  programs  or  increases  in  productivity? 
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10.  How  do  research  expenditures  compare  with 
those  of  competitors? 

11.  How  do  individual  competitors  score  on  some 
of  the  above  pragmatic  tests? 

12.  What  is  the  range  of  the  advanced  product 
planning  program  and  how  good  is  it? 

13.  Does  the  research  division  have  good  rela¬ 
tions  with  the  external  groups  which  it  services? 

14.  Are  there  recognized  innovation  and  crea¬ 
tivity  within  the  research  organizations? 

15.  Are  the  key  people  in  the  R&D  organizations 
strongly  motivated? 

16.  Are  there  effective  techniques  within  the 
research  organization  to  encourage  value  and  produc¬ 
tivity? 

17.  Are  there  enough  people  and  is  there  a  suf¬ 
ficient  variety  of  skills  in  the  research  organiza¬ 
tion  behind  the  key  people?  Is  there  a  personnel 
development  program? 

18.  How  much  confidence  is  there  in  the  indivi¬ 
dual  who  leads  the  research  activity?  [55:409]. 

A  natural  product  can  be  measured  and  subjected  to 
comparison,  and  its  approximate  value  in  the  commercial 
market  can  be  assessed.  .  .  .  However,  .  .  .  the  con¬ 
tribution  cannot  be  ascertained  conclusively  because 
of  inputs  from  other  activities. 

Product  evaluation  is  more  subjective  if  there 
is  no  market  to  act  as  a  value  barometer.  A  good 
illustration  is  a  weapon  purchased  by  the  military. 

Cost  and  performance  can  be  measures,  but  the  bene¬ 
fit  from  the  product  can  only  be  estimated,  because 
its  value  as  a  force  for  deterrence  is  difficult  to 
calculate.  .  .  . 


.  .  .  For  lack  of  a  basis  for  direct  comparison, 
the  value  of  noncommercial  hardware  and  knowledge 
must  be  measured  in  terms  of  mission  requirements  and 
accomplishments  [55:411-412]. 

Roman  also  discussed  input  and  output  determinations, 
but  only  on  the  national  level. 

Roman  listed  the  following  quantifiable  and  quali¬ 
tative  factors  because  "they  do  serve  to  give  a  general  in¬ 
dication  of  evaluation  and  measurement  criteria  which  can  be 
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developed  to  determine  the  efficiency  of  an  R&D  opera¬ 


tion  (55:418)." 

Quantifiable  factors: 

1.  Costs; 

2.  The  percent  of  activity  (sales)  in  new 
projects  or  products; 

3.  The  number  of  new  products  or  projects 
initiated  within  a  given  time  period  as  a  percent 
of  total  product  or  project  mix; 

4.  Profit; 

5.  Personnel  turnover ; 

6.  The  company's  or  organization's  market 
position; 

7.  The  rate  of  growth; 

8 .  Patents  granted-; 

9.  Research  reports. 

Qualitiative  factors : 

1.  The  prestige  and  image  of  the  organiza¬ 
tion; 

a.  Professional; 

b.  Public; 

2.  The  technical  caliber  and  reputation  of  the 
organization's  work; 

3.  The  internal  morale  in  the  organization; 

4.  The  ultimate  benefit  to  be  derived  from  know¬ 
ledge  in  related  work; 

5.  The  retention  of  personnel; 

6.  The  ability,  based  on  technical  reputation, 
to  attract  capable  new  people  to  the  organization; 

7.  The  nature  of  work  in  process  or  contemplated 
as  a  means  of  maintaining  technical  competence  and 
minimizing  professional  obsolescence  [55:418]. 

It  has  been  a  much  more  difficult,  if  not  impos¬ 
sible,  matter  to  construct  realistic,  meaningful 
indexes  or  measurements  for  those  organizations  or 
parts  thereof  (both  government  and  private  industry) 
that  are  primarily  interested  in  research  and  de¬ 
velopment,  advice  and  consultation,  health  and 
safety,  services  of  a  highly  sophisticated  and 
esoteric  nature,  etc.,  the  end  products  of  which 
are  difficult  to  determine,  let  alone  quantify 
[49:656] . 

Newborn  (49:656-657)  discussed  three  measures  deal¬ 
ing  with  the  input  of  labor.  The  first  measure. 
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quantification  of  functions,  consists  of  specifying  unit 
objectives,  identifying  quantifiable  objectives,  and 
identifying  common  quantifications  among  several  organiza¬ 
tions  for  comparison  purposes.  The  second  measure,  work 
sampling,  is  proposed  again  for  group  comparisons.  Per¬ 
formance  appraisal,  the  third  measure,  consists  of  compre¬ 
hensive  judgments  made  of  each  unit's  performance  over  a 
specified  time  period.  Quantitative  terms  are  used  to  ex¬ 
press  qualitative  judgments. 

Szilagyi  and  Wallace  (66:457-458)  further  expand  on 
the  selection  of  an  appraisal  method  as  follows: 

Specifically,  the  problem  for  managers  is  to 
select  a  performance  appraisal  method  that  is  ap¬ 
propriate  given  the  following  ccnsiderations : 

Specific  organizational  and  environmen¬ 
tal  properties,  such  as  technology,  the  de¬ 
sign  of  the  organization,  the  firm's  industry, 
and  other  factors.  ... 

Unique  individual  characteristics  in¬ 
fluencing  performance,  including  specific 
skills  and  abilities  and  motivation  levels. 

The  mix  of  specific  work  behaviors  that 
are  appropriate  given  organizational  and 
individual  considerations. 

The  mix  of  relevant  performance  dimen¬ 
sions,  given  a  consideration  of  the  organi¬ 
zation  and  individuals  involved. 

The  specific  set  of  goals  to  be  achieved 
at  departmental  and  organizational  levels. 

The  above  also  have  application,  of  course,  to  non-R&D  or¬ 
ganizational  environments. 

The  1973  Collier  and  Gee  report  (10:12-17)  focused 


on  evaluating  both  R&D  staff  performance  and  the  potential 
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value  of  completed  projects  (post-evaluation) .  This 

two-step  process  attempts  to  answer  the  following  questions: 

1.  How  well  is  the  research  team  accom¬ 
plishing  the  agreed  upon  objectives? 

2.  How  well  is  the  research  organization 
doing  in  contributing  to  the  company's  commer¬ 
cial  objectives?  [10:12-13]. 

A  weighted  numerical  performance  rating  system  is  suggested 
and  suppored  by  examples.  A  post-evaluation  opportunity 
concept  is  proposed,  where  "opportunity  is  defined  as  the 
size  of  the  market  for  which  the  product  is  both  technically 
and  economically  adequate  (10:16)." 

Collier  (10:16)  stated  the  advantages  of  this 
two-step  post-evaluation  of  R&D  are  as  follows: 

1.  Distinguishes  between  project  selection 
and  post-evaluation; 

2.  Applicable  to  the  short  term; 

3.  Provides  prescriptive  feedback; 

4.  Tends  to  maximize  objectivity  in 
R&D  evaluation; 

5.  Provides  a  more  common  language  for 
research  and  general  management; 

6.  Encourages  communication  between  re¬ 
search  and  the  rest  of  the  organization. 

Collier  expanded  upon  his  proposed  concepts  in  the 
1977  article  "Measuring  the  Performance  of  R&D  Departments" 
(9:30-34) . 

To  evaluate  the  output  of  R&D  operations  you 
must  determine  how  well  predetermined  objectives 
have  been  met  and  the  value  of  business  oppor¬ 
tunities  generated  by  completed  projects  [9:30]. 

Collier  viewed  information  such  as  the  number  of 
dollars  a  company  made  as  a  result  of  R&D  operations  as  an 
improper  method  to  determine  the  productivity  of  an  R&D 
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department  because  this  dollar  value,  depends  on  many  factors 
such  as : 


1.  How  well  management  initially  establishes 
research  objectives ; 

2.  How  well  R&D  interprets  research  objec¬ 
tives  ; 

3.  How  well  the  engineering  department  de¬ 
signs  products  based  on  R&D  generated  technology; 

4.  How  effectively  the  idea  is  translated  into 
production ; 

5.  How  well  marketing  and  sales  personnel 
perform  their  specific  function;  and 

6.  Log  time  between  initial  research  and 
profits. 

What  is  needed,  then,  is  a  system  that  measures  the 
performance  of  the  R&D  department  as  independently  as 
possible  from  the  rest  of  the  company  and  does  so 
quickly  enough  for  the  feedback  to  be  useful  [9:30]. 

Collier  presented  the  argument  that  generally  a 
company  must  go  through  a  three  step  process  to  create  and 
introduce  a  new  or  improved  service  or  oroduct  in  the 
marketplace  (the  responsibility  rests  with  those  departments 
in  parenthesis) : 

1.  Determine  and  agree  upon  what  new  technology 
is  needed  to  achieve  the  company's  goals  (R&D, 
general  management,  marketing,  and  manufacturing) . 

2.  Generate  the  new  technology  agreed  upon 
(R&D)  . 

3.  Apply  the  new  technology  to  the  company's 
operations  in  order  to  achieve  the  overall  goals 
(Engineering,  manufacturing,  sales,  and  R&D). 

A  critical  concept  in  all  of  this  is  that  these 
steps  are  not  simply  additive,  but  utterly  interde¬ 
pendent.  The  productivity  of  the  overall  system  is 
not  the  sum  of  the  individual  productivities  of  each 
step,  but  their  product  [9:30]. 

Collier  (9:31)  examined  the  degree  to  which  the 
R&D  department  achieved  the  agreed-upon  objectives  and  the 
value  of  resulting  business  opportunities  in  an  effort  to 

quantitatively  rate  the  R&D  department  on  an  annual  basis. 
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Collier  first  suggested  a  participative  management  approach 
in  setting  objectives  and  employing  a  weighted  average  method 
for  accumulating  individual  project  rating  scores  to  de¬ 
termine  a  departmental  performance  score.  This  score  com¬ 
pares  the  actual  costs  and  degree  to  which  technology  ob¬ 
jectives  were  achieved  against  previously  established  ex¬ 
pected  costs  and  technology  objectives.  The  objectives 
should  be  challenging  yet  reasonable.  Annual  departmental 
comparisons  may  be  made  and  the  ratings  can  be  used  to  help 
determine  the  distribution  of  bonuses. 

The  second  step  consists  of  annually  evaluating  the 
potential  value  of  the  completed  research  projects.  A  method 
for  calculating  the  value  of  business  opportunity  created  by 
each  project  reaching  the  transition  stage  to  development  is 
proposed.  This  calculated  "return-on-research"  number  is  a 
ratio  of  the  amount  of  opportunity  produced  for  every  research 
dollar  spent  -  a  "return  on  research"  index  of  producing 
opportunity  efficiency.  The  index  allows  annual  company, 
department,  and  subgroup  comparisons  to  be  made.  The  diffi¬ 
cult  calculation  is  in  determining  business  opportunity. 

The  numbers  that  go  into  a  business  opportunity 
calculation,  although  sometimes  hard  to  obtain,  are 
mostly  objective  numbers  upon  which  all  parts  of  the 
company  should  be  able  to  agree.  Furthermore,  if 
you  set  up  rules  on  how  to  use  the  numbers  in  calcu¬ 
lations,  and  apply  them  uniformly,  project  by  pro¬ 
ject  and  year  by  year,  the  results  should  be  useful 
in  making  comparisons .... 

In  terms  of  what  profits  the  company  might  ac¬ 
tually  realize  from  the  technology,  obviously  the 
business  opportunity  numbers  are  unrealistically 
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high.  For  example ,  the  price  of  the  new  develop¬ 
ment  would  have  to  be  set  below  the  value-in-use 
price,  to  give  the  potential  customer  a  driving 
reason  to  switch  to  your  product.  It  is  also 
unrealistic  to  expect  you  could  ever  reach  100% 
market  share.  So  these  numbers  have  to  be  recog¬ 
nized  as  absolute,  "laboratory",  values,  which 
would  lack  credibility  for  use  in  discussions 
with  other  parts  of  the  company — especially  if 
used  to  help  convince  general  or  marketing  manage¬ 
ment  of  the  value  of  research,  or  to  persuade  them 
to  prosecute  a  particular  project  farther  [9:32]. 

Collier  provided  an  example  to  follow  and  four  recommended 

rules  for  using  this  method,  to  summarize  with: 

There  is  still  much  work  to  be  done  before  I 
am  fully  satisfied  with  the  evaluation  technique, 
but  I  believe  it  is  becoming  an  increasingly 
valuable  tool  in  the  research  director's  manage¬ 
ment  armamentarium  [9:34]. 

Seitz  (59:26-83)  discussed  the  following  five 
characteristics  of  productive  companies: 

1.  Effective  use  of  available  capital,- 

2 .  A  well-managed  research  and  development 
effort; 

3.  Constructive  util  nation  of  human  re¬ 
sources} 

4.  An  organization  structure  that  stimulates 
productivity  gains; 

5.  Adherence  to  a  well-constructed  and  well- 
communicated  management  philosophy  or  ethic 
[59:26]  . 

Seitz  (59:26-28)  stated  that  recent  research  indi¬ 
cates  that  companies  may  increase  their  return  on  R&D  by: 

1.  Improving  project  selection  and  management; 

2.  Encouraging  managers  to  accept  projects  with 
higher  risk  and  payout; 

3.  Strengthening  the  ties  between  marketing  and 
R&D  departments . 

Faust  (22:39-42)  focused  on  the  concept  that  today, 

more  than  ever,  scientists  must  communicate  with  senior 
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management  and  favorably  influence  their  understanding  of 
and  their  financial  support  of  R&D.  The  following  subtopics 
in  pharmaceutical  R&D  are  discussed: 

1.  Assessing  the  impact  of  legislation  on  the 
cost  of  research; 

2.  Determining  the  cost  of  drug  development; 

3.  Measuring  the  productivity  of  research; 

4.  Identifying  and  communicating  research 
needs/strategies ; 

5.  Conveying  incentives  to  invest  in  research 
[22:38-42]. 

In  measuring  the  productivity  of  research  in  the 
pharmeceutical  community,  Faust  (22:40-41)  suggested  several 
possibilities: 

1.  New  single  chemical  entities  (NCEs)  intro¬ 
duced  per  research  investments; 

2.  The  contribution  of  NCE’s  to  sales  and 
profits; 

3.  The  importance  of  NCEs; 

4.  "Defensive"  research  investments  to 
comply  with  changing  regulatory  requirements; 

5.  The  contribution  to  the  storehouse  of 
pharmeceutical  and  biomedical  knowledge. 

Mali  (41:100-102)  suggested  that  "Evaluations  using 
checklist  indicators  can,  in  an  indirect  way,  measure  pro¬ 
ductivity  by  specifying  the  actions  to  be  taken  that  can  mea¬ 
sure  both  performance  effectiveness  and  resource  efficiency 
(41:100)."  Several  examples  are  given,  including  one  for 
an  R&D  employee  which  lists  20  indicators  as  follows: 

1.  Looks  for  improvement; 

2.  Has  record  of  accomplishments; 

3.  Learns  a  new  assignment  quickly; 

4.  Has  a  strong  will  to  work,  keep  busy; 

5.  Has  good  work  habits; 

6.  Has  a  strong  sense  of  commitment  to 
completing  work; 

7.  Is  cooperative  in  teamwork; 

8.  Is  open  to  ideas  and  listens  well; 

9.  Use^  time  effectively; 

10.  Takes  initiative  to  do  things; 
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11.  Is  cost  minded; 

12 .  Has  a  strong  sense  of  urgency; 

13.  Gets  satisfaction  from  a  job  well 
done; 

14.  Contributes  beyond  what  is  expected; 

15.  Knows  the  job  well; 

16.  Sees  things  to  be  done  and  takes 
action; 

17.  Is  considered  valuable  by  supervisor; 

18.  Interacts  effectively  with  other 
people; 


19.  Understands  organizations  and  their 
objectives; 

20.  Believes  in  a  fair  day's  pay  [41:101-102]. 

The  employee  productivity  index  then  equals  the  ratio  of  in¬ 
dicators  observed  with  the  employee/ total  number  of  indica¬ 
tors.  This  is  one  method  of  assigning  a  quantitative  value 
to  qualitative  criteria,  and  may  be  expanded  to  projects 
and  groups. 

Griliches  (27:92-116)  outlined  a  production  function 
approach  to  estimate  the  returns  to  R&D  on  a  national  level 
and  relates  it  to  economic  growth.  The  measurement  of  output 
in  R&D  intensive  industries  and  the  definition  and  measure¬ 
ment  of  the  stock  of  R&D  capital  are  also  discussed. 

The  production  function  describes  the  relationship 
between  the  final  output  and  the  various  inputs.  The  defini¬ 
tion  and  measurement  of  the  stock  of  R&D  capital  leads  to  a 
discussion  and  modeling  of  R&D  spillover  effects.  Industrial 
sectors  and  not  specific  company  models  are  addressed.  A 
significant  reference  list  is  provided  to  further  pursue 
this  viewpoint. 

Hamelink  and  Hamelink  (28:88-89)  provided  a  model  for 
a  performance  appraisal  which  Roman  (55:391)  previously 
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pointed  out  is  the  starting  point  for  R&D  evaluation  mea¬ 


surement. 

Consider  assigning  a  numerical  value  to  each 
key  element  reflecting  its  relative  importance  to 
the  job  purpose.  Assign  also  a  number  to  the  per¬ 
formance  standard  and  one  to  the  actual  performance 
based  on  the  relative  difficulty  of  accomplishing 
the  particular  key  element  and  percentage  of  ac¬ 
complishment  of  that  key  element  [28:88]. 

If  actual  performance  (AP)  less  the  performance 
standard  (PS)  is  positive,  then  the  performance  is  above 
standard  (AS).  If  AP-PS  is  zero,  then  the  performance  is 
standard  (S) .  If  AP-PS  is  negative,  then  the  performance 
is  below  standard  (BS)  or  deficient  (D) .  Hamelink  and 
Hamelink  further  develop  the  analysis  to  yield  vector  values 
and  a  Root  Mean  Square  (RMS)  value  in  an  effort  to  reduce 
the  subjectivity  of  performance  appraisals  and  meet  the  cri¬ 
teria  of  fairness  and  accurate  employee  representation. 

In  1980,  Karger  and  Murdick  (34:415-430)  addressed 
the  measurement  of  engineering  and  research  (E&R) .  They 
state  that  a  measurement  and  evaluation  system  must  be  based 
on  the  following  principles: 

Measurement  and  evaluation  should  be: 

1.  Related  to  company  goals; 

2.  Based  on  long-and  short-range  technical 
plans; 

3.  Taken  into  account  when  relating  the 
personal  goals  of  the  individual  professional 

to  the  organizational  goals  discussed  in  earlier 
chapters; 

4.  Partially  based  on  standards  related  to 
the  industry,  the  nation  in  which  the  organiza¬ 
tion  is  located,  and  on  similar  performance 
standards  in  other  nations; 

5.  Readily  understandable  by  everyone; 
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6.  Limited  to  key  points  and  events,  like 
objectives  and  project  milestones; 

7.  Objective  and  quantitative,  if  possible; 

8.  Able  to  detect  variances  and  trouble  spots 
quickly; 

9.  Economical; 

10.  Provide  a  stimulus  to  action  [34:416]. 

To  summarize  the  basic  Karger  and  Murdick  approaches  to 
measuring  E&R,  criteria  in  the  form  of  objectives  or  stan¬ 
dards  must  be  established  and  compared  with  actual  perfor¬ 
mance.  The  following  factors  are  measurable  for  this  purpose 

1.  Inputs: 

a .  Money ; 

b.  Personnel; 

c.  Facilities; 

d.  Time  for  major  company  program  or  growth; 

2.  Process  (professional  performance,  managerial 
and/or  technical)  against  internally  generated  stan¬ 
dards  some  of  which  can  and  should  reflect  industry 
standards,  represented  by  goals  and  objectives; 

3.  Output  as  a  contribution  to  the  business, 

4.  Valua-in-exchange  (this  is  the  in-house 
versus  purchase  comparison)  [34:418]. 

The  quality  of  E&R  may  be  indirectly  determined  from 
an  indexing  of  patents,  publications,  and  citations  of  work 
or  publications  and  "some  qualitative  measurement  should  al¬ 
ways  be  used  (34:419)."  E&R  may  be  measured  at  each  manage¬ 
ment  level  for  internal  control  (activity  direction  at  the 
corresponding  levels  and  top  management  coordination.) 
indexes  of  input  quality  are  given  as  follows: 

1.  Ratio  of  clerical  support  to  engineers 
and  scientists; 

2.  Square  feet  of  lab  and  office  space  per 
engineer/scientist; 

3.  Performance  cost  per  engineer/scientist; 

4.  Average  age  of  research  equipment  and 
instruments; 

5.  E&R  expenditures/sales  per  year; 

6.  E&R  expenditures/profits  per  year; 
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7.  E&R  expenditures/sales  versus  national 
E&R  expenditures/GNP  ; 

8.  Dollar  backlog  of  projects; 

9.  Man-years  backlog  of  projects; 

10.  Average  salary  per  engineer/scientist; 

11.  Average  salary  per  manager  in  E&R  [34:420]. 

Current  comparison  data  for  industry  and  the  national 
average  may  be  obtained  from  the  Chamber  of  Commerce  of 
the  United  States,  Washington,  D.C. 

Individual  output  and  efficiency  equations  were  dis¬ 
cussed  and  enhanced  by  time-difficulty  and  quality  of  work 
versus  rate  of  work  curves.  Some  criteria  related  to  de¬ 
termining  how  much  a  group  contributes  to  achieving  E&R 
objectives  are  as  follows: 

1.  Actual  development  time  for  a  project  com¬ 
pared  with  a  projected  time  based  on  standards  es¬ 
tablished  for  the  class  of  product  involved; 

2.  Percentage  of  dollar  change  (plus  or  minus) 
in  average  yearly  development  cost  per  project  by 
class; 

3.  Number  of  new  and/or  products  improved  per 
year  (or  month) ; 

4.  Number  of  new  products  completed  per  year 
(or  month) ; 

5.  Change  (plus  or  minus)  in  the  average  yearly 
pay-off  per  project  for  a  class  of  products; 

6.  Change  (plus  or  minus)  in  total  pay-off  or 
profit  earned  on  products  designed  by  the  group; 

7.  Square  feet  of  new  design  drawings  per 
month  for  a  section; 

8.  Estimated  or  actual  total  pay-off  for  all 
products  the  section  has  developed; 

9.  Actual  total  pay-off  for  all  products  de¬ 
veloped  and  put  in  use  on  the  market; 

10.  Development  cost  per  person; 

11.  Development  project  cost  per  engineer  or 
scientist ; 

12.  The  trend  of  the  ratio  of  the  cost  of  the 
E&R  unit’s  cost  to  the  sales  volume  of  its  product 
line,  providing  this  is  applicable  [34:424], 
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Most  of  the  group  measurements  and  the  following  measure¬ 
ment  criteria  can  be  applied  to  the  total  E&R  organiza¬ 
tion: 


1.  How  well  is  E&R  doing  in  relation  to  com¬ 
petitors  regarding  the  improvement  of  products  and 
the  finding  of  new  ideas? 

2.  Are  the  products  reliable,  easily  produced, 
and  maintainable? 

3.  Do  the  products  fit  the  market  needs , a? 
interpreted  by  Marketing? 

4.  Does  the  E&R  organization  always  have  a  back¬ 
log  of  potentially  profitable  projects  which  it  has 
proposed? 

5.  Is  the  E&R  organization  building  a  reservoir 
of  knowledge  and  attracting  talented  new  people? 

6 .  Are  good  technical  reports  of  progress 
issued  periodically? 

7.  Are  costs  of  E&R  controlled  reasonably 
well? 

8.  Is  there  a  steady  flow  of  patents  being 
issued? 

9.  Is  the  company  forced  to  pay  royalties 
because  E&R  failed  to  invent  and  patent  needed  in¬ 
ventions  . 

10.  Is  the  research  group  "lucky"  enough  to 
come  up  with  a  very  valuable  discovery  from  time 
to  time? 

11.  Have  the  products  been  so  well  designed 
that  customer  complaints  are  at  a  bare  minimum? 

12.  Has  the  product  design  protected  the 
company  from  liability  suits?  [34:426-427]. 

Karger  and  Murdick  included  a  chapter  bibliography 
and  summarize  the  measurement  of  E&R  by  stating  many  qualita¬ 
tive  and  quantitative  criteria  may  be  used  to  evaluate  E&R 
and  the  E&R  managers  should  be  receptive  to  applicable  eva¬ 
luation  criteria. 

The  fact  that  the  tools  may  be  poor  does  not 
negate  management's  responsibility  (including 
the  manager  of  E&R)  to  measure  engineering  and 
research  as  best  it  can  [34:429]. 

The  proposed  theoretical  methodologies,  indicators, 
and  needs  for  R&D  productivity  measurements  have  been 
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chronologically  presented.  The  next  section  identifies 
the  empirical  studies  which  have  been  conducted  since 
1960  to  reveal  the  consistencies,  contradictions,  and 
novelties  between  the  theoretical  and  empirical  literature. 


Empirical  Review 

Since  technological  change  is  a  chief  cause  of 
productivity  advance,  measures  of  innovational  ac¬ 
tivity  in  the  various  industries  should  be  signi¬ 
ficantly  correlated  with  relative  productivity 
changes.  But  even  if  we  could  catalogue  all  the 
innovations  made  by  the  firms  of  each  industry  in 
successive  periods,  further  difficulty  would  be 
met  in  trying  to  weight  each  in  accordance  with 
its  relative  importance.  At  best,  only  indirect 
measures  of  innovational  activity  such  as  the 
number  of  patents  issued,  are  possible  [36:182]. 

Statements  similar  to  the  Kendrick  quote  above 
are  representative  of  the  thoughts  concerning  R&D  activities 
in  the  late  1950s  and  1960s.  Quinn  (55:15-42)  discussed 
and  provided  examples  for  each  of  the  following  three  ap¬ 
proaches  to  measuring  the  productivity  of  research  (current 
as  of  1959);  the  quantitative,  qualitative,  and  integrated 
approaches.  Quinn  interviewed  58  executives  from  28  com¬ 
panies  to  identify  yardsticks  of  industrial  research  based 
on  this  categorization  (55:209-219). 

The  quantitative  approach  attempts  to  evaluate  the 
research  program  and  its  contributions  with  the  aid  of 
a  mathematical  formula  using  one  or  more  of  the  following 
factors: 


1.  Profits  from  research-created  new  products; 

2.  Profits  from  cost-reducing  new  or  improved 
processes,  methods,  or  raw  materials  based  upon  re¬ 
search; 

3.  Profits  from  improved  products; 
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4.  Savings  from  royalty  payments  avoided; 

5.  Income  from  royalty  payments  received; 

6.  Miscellaneous  profit  contributions 
resulting  from  general  goodwill  created  by  the 
research  establishment; 

7.  The  investment  in  the  R&D  program  on  the 
total  investment  to  bring  research  technology  to 
commercial  fruition  [55:16]. 

Nearly  every  company  Quinn  contacted  reported  that  some 
form  of  the  quantitative  evaluation  approach  had  been  ex¬ 
perimented  with.  Both  favorable  and  unfavorable  results 
were  reported.  Of  the  companies  contacted,  14.8%  relied 
heavily  on  quantitative  devices,  48.2%  used  some  quantita¬ 
tive  devices  but  depend  primarily  on  individual  or  group 
judgment,  and  37.0%  relied  little  or  not  at  all  on  quanti¬ 
tative  evaluation  devices  (55:22). 

The  qualitative  approach  consists  of  "broad  com¬ 
posite  management  judgments"  made  by  committees  or  individual 
executives.  Comments  supporting  the  qualitative  approach 
reflect  the  following  thoughts  (55:22-23) : 

1.  Quantitatively,  research  contributions 
cannot  be  differentiated  from  the  contributions 
of  other  functional  groups; 

2.  R&D  pays  for  itself.  We  don't  need 
figures  to  tell  us  that; 

3.  Attempts  to  quantify  R&D  are  not  fruit¬ 
ful  at  this  time ; 

4.  There  is  no  method  of  evaluation  that 
does  not  ultimately  rely  on  the  judgment  of  a 
few  individuals . 

In  the  qualitative  approach,  personal  appraisals  are  pyramided 
to  top  management  with  each  higher  management  level  relying 
on  the  qualitative  evaluation  of  the  next  succeeding  lower 
level.  The  top  technical  executive  then  explains  and 
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interprets  the  technical  progress  to  the  top  level  non¬ 
technical  president,  operating  committee,  or  board.  A 
great  deal  of  faith  is  placed  in  the  top  technical  execu¬ 
tive. 


The  integrated  approach  is  an  evaluation  of  how 
well  research  integrates  with  the  operating  groups  of  the 
company  using  both  qualitative  and  quantitative  techniques. 

In  other  words,  how  well  does  the  research  activity  support 
broader  company  operating  objectives,  since  research  is  an 
integral  part  of  the  whole  industrial  company. 

The  research  group  should  be  held  responsible 
for  providing  required  technology  for  the  company 
just  as  sales  is  held  responsible  for  supplying  it 
with  customers.  The  objectives  of  research,  there¬ 
fore,  can  only  be  established  in  terms  of  the  needs 
of  the  particular  company  which  is  supporting  the 
research  organization.  Once  these  objections  are 
recognized,  they  define  the  responsibilities  of 
the  research  organization  and  form  the  basis  for 
measuring  its  effectiveness  t55:31]. 

The  integrated  approach  attempts  to  determine  whether 
a  research  organization  is  producing  results  consistent 
with  company  objectives  at  the  right  times  and  in  the  right 
quantities  with  the  following  types  of  interrogative 
yardsticks : 


1.  Does  the  research  organization  know  just 
what  its  company  needs? 

2.  Is  the  research  organization  creating  the 
needed  technology? 

3.  Is  the  research  organization  helping  to 
get  research  results  applied? 

4.  Are  the  research  organization's  operations 
efficient? 

5.  Is  the  research  management  an  effective  part 
of  the  executive  team?  [55:32]. 


42 


Several  interviews  stated  that  the  integrated  approach 
was  a  primary  criterion  for  evaluating  research  manage¬ 
ment,  but  no  one  company  relied  solely  on  the  integrated 
approach. 

In  summarizing,  Quinn  states: 

There  can  be  no  single  quantitative  measure 
which  adequately  appraises  all  important  facets  of 
the  research  effort  because  there  is  not  even  a 
package  of  quantitative  devices  which  cam  be  use¬ 
fully  applied  to  all  phases  of  the  research  pro¬ 
gram.  Current  long-range  research  results,  re¬ 
search  efficiency,  research  quality,  and  intangible 
research  results  defy  quantitative  appraisal. 

In  the  appraisal  of  certain  phases  of  research, 
therefore,  there  is  no  alternative  to  the  well- 
reasoned  and  experienced  judgments  of  individuals 
or  groups .... 

On  the  other  hand,  quantitative  techniques 
offer  a  degree  of  objectivity,  which  cannot  be  ob¬ 
tained  qualitatively.  Consequently ,  quantitative 
measures  are  recommended  where  they  are  not  simply 
pedantic  and  where  they  do  not  measure  with  rea¬ 
sonable  accuracy  what  they  intend  to  measure. 

Throughout  the  research  evaluation  process , 
the  necessity  for  proper  integration  with  the  com¬ 
pany's  operation  must  be  accepted  and  judgments 
concerning  the  adequacy  of  integration  facilitated 
[55:41-42]  . 

Quinn  uses  the  term  technical  evaluation  to  signify 
the  evaluation  of  the  efficiency  and  quality  of  re¬ 
search  performance. 

Technical  evaluation  involves  making  judgments 
concerning  the  period's  technical  accomplishment 
without  regard  to  the  ultimate  commercial  conse¬ 
quences  of  the  technology  created.  .  .  . 

Technical  evaluation  appraises  (1)  the  ef¬ 
ficiency  with  which  researchers  achieved  the 
period's  planned  technological  goals,  (2)  the 
degree  of  creativity  their  work  demonstrated, 
and  (3)  the  technical  proficiency,  or  skill, 
which  their  work  evidenced.  .  .  . 
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Efficiency  evaluation  relates  to  the  actual 
cost  of  performing  research  to  some  standard  which 
says  how  much  it  should  have  cost  [55:62-63], 

Volume,  efficiency,  price,  and  organizational 
variances  are  discussed  in  relation  to  efficiency  evalua¬ 
tions  (55:74-78),  and  efficiency  evaluations  are  closely 
interrelated  with  quality  evaluations  of  research.  Ac¬ 
cording  to  Quinn,  creativity  and  technical  proficiency 
are  the  two  factors  which  determine  the  quality  of  research 
results.  But  "standards  for  judging  creativity  and  techni¬ 
cal  proficiency  are  extremely  obscure  and  notably  subjec¬ 
tive  (55:79).”  The  research  executives  polled  indicated 
that  rankings  are  used  to  accurately  indicate  personnel 
creativity  and  technical  proficiency  in  an  effort  to  tho¬ 
roughly  appraise  research  output.  Quality  valuations  are 
pyramided  to  top  management  in  the  same  manner  described 
under  the  qualitative  approach. 

"Commercial  success  ...  is  not  a  useful  criterion 
for  evaluating  the  quality  of  research  results  (55:82)." 

This  statement  is  supported  by  examples  where  the  end  result 
of  a  specific  research  was  the  extension  of  knowledge  or 
the  demonstration  that  an  approach  to  a  problem  will  never 
be  feasible,  yet  the  quality  of  the  research  may  be  very 
high. 

Quinn  further  stated  that  factors  affecting  the 
use  of  patents  as  criteria  for  quality  evaluations  include: 

1.  Time  lags  between  invention  and  patent 
issue  dates ; 
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2.  Actual  patent  value  may  be  determined 
years  later  in  a  commercial  application; 

3.  Patent  value  is  not  indicated  by  the 
number  of  patents  per  time  period ; 

4.  Company  patent  policy  (trade  secrets); 

5.  Subjective  judgments  are  required  [55:83-85]. 

Similarly,  the  interviews  indicated  that  the  use  of 
publications  as  standards  for  quality  evaluation  has  several 
factors  to  be  considered  such  as: 

1.  Company  publications  policy; 

2.  National  (and  company)  security; 

3.  Varying  quality  of  the  publications; 

4.  More  appropriately  used  to  evaluate 
an  individual  [55 : 86-87]  . 

Few  considered  the  publication  rate  to  be  an  accurate  or 
usable  creativity  measure.  "No  operating  or  functional 
manager  contacted  reported  any  use  of  publications  as  a 
check  on  the  quality  of  research  results  (55 : 87) . " 

In  1961,  Taylor  et  al  (68:1-3)  intensely  interviewed 
215  physical  scientists  working  in  15  different  laboratories 
at  two  Air  Force  basic  research  centers  in  an  effort  to  de¬ 
termine  methods  to  predict  productivity  and  creativity. 

The  term  contribution  is  used  to  mean  productive  and  creative 
acts.  This  three  year  project  consisted  of: 

1.  a  large  interview  activity; 

2.  a  study  to  deal  with  the  contributions 
of  scientists  (ID  criteria) ; 

3.  developing  measures  of  psychological  charac¬ 
teristics  and  validate  these  measures  against  the 
criteria  in  2. 

Each  two  year  interview  pursued  the  following  issues: 

(a)  types  of  scientists,  (b)  kinds  of  work  done, 

(c)  personal  background  information,  (d)  probable 
criteria  of  scientific  productivity  and  creativity, 
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(e)  suggested  ways  for  recording  and  scoring  be¬ 
havior  on  such  criteria,  {f)  nature  of  the  talent 
in  ones  own  laboratory,  (g)  ways  of  predicting 
success  in  science,  (h)  conditions  which  influence 
scientific  contributions,  and  (i)  other  questions 
which  seemed  appropriate  in  each  interview  situa¬ 
tion  [68:2]  . 

Resulting  interview  responses  "...  implied  that  working 
conditions  have  important  effects  on  the  quantity  and 
quality  of  scientists'  contributions  .  .  .  (68:3)"  and 
included  "...  statements  related  to  the  primary  goals  of 
the  project  (68:3)." 

An  analysis  of  the  interview  statements  suggested 
the  following  kinds  of  criteria: 

1.  Ratings  by  others  of  the  scientist  and  his 
scientific  products.  Typical  of  such  criteria  were 
subjective  ratings  of  productivity  by  one's  supervi¬ 
sors,  peers,  and  a  research  psychologist;  a  complex 
quality  and  quantity  evaluation  of  publications  and 
research  reports;  and  the  collection  of  quality  and 
quantity  information  about  patents  and  invention 
disclosures. 

2.  Collection  of  objective  facts  about  each 
scientist  from  Air  Force  records.  The  number  of 
publications  one  had  produced  during  a  given  two- 
year  period  with  the  organization;  the  number  of 
oatents ,  invention  disclosures,  and  evaluations 
of  them;  number  of  contracts  monitored;  and  or¬ 
ganizational  background  history  of  each  scientist 
were  the  major  kinds  of  probable  criterion  infor¬ 
mation  available  from  these  records  [68:3]. 

Taylor  required  the  following  15  categories  to 
adequately  categorize  the  52  contribution  scores  of  166 
physical  scientists: 

A.  Productivity  in  written  scientific  work; 

B.  Recent  quantity  of  research  reports; 

C.  Quality  (without  originality)  of  research 
reports ; 

D.  Originality  of  written  work; 

E.  Scientific  and  professional  society  member¬ 
ships  ; 
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F.  Actual  quantity  of  work  output,  as  judged 
by  peers,  supervisors,  and  laboratory  chiefs; 

G.  Creativity  rating  by  laboratory  chief; 

H.  Overall  performance; 

I.  Likeableness  as  an  effective  member  of 
the  research  group; 

j.  Visibility  of  the  scientist; 

K.  Recognition  for  organizational  contribu¬ 
tion; 

L.  Status-seeking,  "organizational-man" 
tendencies ; 

M.  Current  organizational  status; 

N.  Contract  monitoring; 

O.  Total  years  of  work  experience  [68:10,31]. 

The  validation  study  was  completed  on  107  of  the  above 
scientists,  using  103  predictor  scores  and  17  criterion 
scores.  .  . 

For  every  criterion.,  there  were  several  predictors 
with  low  to  moderate  validity.  If  an  independent  sample 
of  scientists  were  available,  effectiveness  of  best  com¬ 
binations  of  predictors  could  be  tested.  The  most  pre¬ 
dictable  criteria,  in  terms  of  number  of  valid  scores 
were  likeableness  as  a  member  of  the  research  team, 
professional  society  membership,  current  organizational 
status,  judged  work  output,  supervisory  ratings  on 
overall  performance,  and  peer  rankings  on  overall  pro¬ 
ductivity. 

The  psychological  scores  that  were  valid  for  the 
largest  number  of  these  criteria  were  those  for  crea¬ 
tivity,  inner  directedness ,  discrimination  of  value, 
professional  self  confidence,  cognition,  desire  for 
principles,  drive,  self-sufficiency,  flexibility,  in¬ 
dependence,  intuition,  aspiration  in  quantity  of  re¬ 
search  reports,  aspirations  in  theoretical  contribu¬ 
tions,  aspiration  in  high  level  of  original  work,  and 
intellectual  thoroughness  [68:31-32]. 

Mullins  (46:52-57)  used  a  questionnaire,  a  vocabu¬ 
lary  test,  and  nine  tests  of  the  Guilford  Creativity  Battery 
to  survey  131  scientists  in  an  effort  to  predict  creativity. 

Word  association,  unusual  uses,  common  situations  and  plot 

V. 

titles  were  found  to  be  significantly  related  to  the  super¬ 
visors'  rating  criterion.  Unusual  uses,  common  situations, 
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plot  titles,  consequences,  and  brick  uses  were  found  to 
be  significantly  related  to  publications,  the  second  cre¬ 
ativity  criterion.  None  of  the  predictor  variables  correlated 
significantly  with  both  criteria  and  the  two  criteria  were 

found  to  not  be  significantly  related  to  each  other. 

The  results  of  Villers  (71:65-74)  interview  with  269 

individuals  in  34  companies  indicated  a  strong  utilization 
of  PERT  and  GANTT  charts  for  relating  technical  progress  to 
expenditures.  Since  most  research  is  programmatic  (sche¬ 
duled?  conducted  by  using  a  written  plan),  "...  the  actual 
expenditures  can  be  clearly  related  to  the  expected  techni¬ 
cal  accomplishments  (71:77)." 

According  to  Villers  (71:77-79),  the  following  four 
steps  were  used  by  many  companies  which  allow  researchers  to 
evaluate  their  technical  progress : 

1.  Periodic  review  of  technical  progress; 

2.  Relate  the  progress  to  the  schedule; 

3.  Evaluate  the  impact  of  any  delay; 

4.  Select  the  optimum  alternative  after 
the  impact  of  the  delay  has  been  determined; 

5.  Time  and  money  variance  analysis. 

Villers  measured  research  input  by  money  allocated 
and  research  output  by  technical  accomplishment  (71:6).  The 
following  four  points  were  emphasized  as  necessary  for  suc¬ 
cessful  R&D  planning  and  control: 

1.  Tangible  cost  vs.  potential  benefit; 

2.  Potential  language  benefits  vs.  actual 
short-range  expenditures? 

3.  Encouraging  research  people  to  understand 
the  need  for  profit-making? 

4.  Encouraging  management  to  recognize  the 
complexity  of  R&D  operations  [71:8-9]. 
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Villers  (71:108)  found  that  top  management  needs 
quantitative  measurements  of  the  benefits  of  r&d  activities 
to  decide  whether  or  not  specific  R&D  projects  should  be  pur¬ 
sued.  The  costs  are  readily  determinable,  but  the  benefits 
or  contributions  are  not.  "The  efficiency  of  research  acti¬ 
vities  cannot  be  measured  by  standards  such  as  the  rate  of 
production  per  man-hour  or  the  sales  volume  during  a  certain 
period  (71:108)."  The  following  four  classes  of  objections 
to  quantitatively  measure  the  benefits  of  R&D  activities  pro¬ 
vide  support  for  qualitatively  evaluating  R&D  activities: 

1.  Rivalry  between  the  various  corporate  func¬ 
tions  (i.e.,  production  and  marketing); 

2.  Rivalry  within  the  R&D  personnel; 

3.  Risk  of  ignoring  some  factors  (competition); 

4.  Psychological  problems  (pressure  to  per¬ 
form)  [71:109-110]. 

Villers  noted  that,  in  some  companies,  qualitative 
reports  (standardized  forms  excluding  cost  allocation)  are 
submitted  to  top  management  to  summarize  technical  accomplish¬ 
ments  and  their  benefits;  to  state  the  current  status  of  spe¬ 
cific  projects;  to  supply  figures  that  are  related  in  some 
way  to  the  efficiency  of  R&D  activities.  Some  of  the  factors 
that  were  considered  are  as  follows: 

1.  Number  of  patents  (or  company's  number 
of  patents  vs.  competitor's  number  of  patents); 

2.  Number  of  disclosures  presented  to  the 
patent  attorney  by  the  research  people; 

3.  List  of  scientific  papers  presented 
outside; 

4.  Royalties  received  from  other  companies; 

5.  Dollars  of  sales  of  new  products:  the  trend 
of  the  new-product  sales  measured  as  a  percentage 
of  the  total  sales  for  the  year; 

6.  Increase  of  the  company's  share  of  the  mar¬ 
ket  for  specific  product  lines; 
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7.  Cost-saving  accomplishments; 

8.  Detailed  review  of  specific  areas 

where  technological  accomplishments  have  proved  to 
be  especially  beneficial  to  the  company; 

9.  Evaluation  of  the  rank  of  the  company  on 
the  market  [71:112]. 

Seiler  (58:189-198)  surveyed  116  companies  in  1965 
and  reports  that  there  were  four  major  methods  used  in  in¬ 
dustry  to  evaluate  research: 

1.  Ratios; 

2.  Cash  flows; 

3.  Project  cost  appraisals; 

4.  Research  income  statements; 

Seiler  concisely  reviews  each  method,  articulating  several 
examples,  deficiencies,  and  benefits. 

"Evaluation  is  a  part  of  planning  and  should  be 
undertaken  in  the  light  of  both  corporate  and  re¬ 
search  objections.  Any  effective  evaluation  sys¬ 
tem  must  provide  a  means  of  measuring  effectiveness 
and  efficiency,  but  in  the  case  of  R&D  only  broad 
guages  covering  fairly  long  time  spans  are  meaning¬ 
ful  [58:197-198]. 

Eidusun  (19:57-63)  performed  a  follow-up  study  in 
1964  of  39  research  scientists  to  compare  the  rates  of  pro¬ 
ductivity  with  the  results  of  a  1958  study.  "The  number 
of  scientific  publications  provided  the  index  of  produc¬ 
tivity,  and  was  obtained  from  a  detailed  curriculum  vita 
submitted  by  each  subject  (19:58)."  The  results  show  that 
in  general,  research  scientists: 

maintain  a  stable  productivity  rate  or  even 
increase  their  previous  performance  level. 

This  pattern  holds  for  each  age  group  up  to 
the  age  of  60.  Only  when  scientists  leave 
research  for  administration  or  teaching,  or 
leave  the  university  for  industry,  do  de¬ 
creases  in  productivity  become  apparent 
[19:63] . 
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Productivity  remained  constant  over  time  when  publications 
were  used  as  the  index. 


Burgess  (7:85)  first  categorized  Rome  Air  Development 
Center  (RADC)  in-house  R&D  into  the  creative,  non-creative , 
and  contract  support  classifications  indicated  in  Table  3-2. 


TABLE  3-2 

RADC  R&D  SPECTRUM  OF  DIRECT  ENGINEERING  ACTIVITY 


IN-HOUSE  (CREATIVE) 


1.  Original  ideas  with  bench-work  experimentation. 

2.  Original  study  with  publication  as  end  item. 

3.  Suggested  ideas  with  bench-work  experimentation. 

4.  Experiments  with  unique  equipment  to  derive  new 
techniques  or  general  knowledge. 

5.  Engineering  activity  to  adapt  state-of-art  tech¬ 
niques  to  hardware . 


IN-HOUSE  (NON-CREATIVE 


6.  Technical  activity  in  support  of  scientific 
committees,  symposia,  ad  hoc  groups. 

7.  state-of-art  experiments. 

8.  Investigations  utilizing  government  facilities 
against  operational  equipments. 

9.  Formulation  of  technical  programs. 

LO.  Analyses  and  consultations  required  for  opera¬ 
tional  developments  (not  identifiable  with  any 
contract) . 

LI.  Test  design  and  evaluation  of  equipment. 

L2.  Design  development  prior  to  contracting. 


ZONTRACT  SUPPORT  (BASIC  RESEARCH  THROUGH  SYSTEMS  DEVELOPMENT) 

L3.  Evaluation  of  unsolicited  proposals. 

L4.  Preparation  of  work  statements. 

L5 .  Engineering  support  to  procurement  during  nego¬ 
tiations. 

L6.  Contract  monitoring. 

L7.  Acceptance  tests  exclusive  of  11,  above. 
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Burgess  emphasized  that  different  types  of  R&D  labora¬ 
tories  (government,  industry,  and  university)  perform 
different  work  requiring  different  yardsticks  for  evalua¬ 
tion  (site  specific) . 

Industry's  laboratory  program  can  be  evaluated 
in  terms  of  dollars — new  products  that  create  new 
business  and  profits  for  the  corporation.  How 
many  times  have  we  heard  a  company  brag  that  50 
percent — or  60  percent — or  75  percent  of  its  pro¬ 
ducts  were  introduced  during  the  previous  three 
to  five  years!  The  university  laboratory  program 
can  be  evaluated  in  terms  of  new  M.S.'s  and  Ph.D.'s, 
or  new  knowledge.  However,  the  DOD  laboratory  must 
measure  its  effectiveness  against  its  contributions 
to  the  technological  war  in  which  our  country  was 
engaged. 

This  leads  to  the  first  important  factor  for 
evaluation:  A  laboratory,  any  laboratory,  must  be 
evaluated  against  the  long-range  objectives  and 
goals  of  its  parent  organization  [7:86]. 

The  second  important  factor  is  stated  as: 

The  laboratory  should  stay  within  its  assigned 
mission,  contribute  increases  in  the  state-of- 
the-art,  and  translate  these  increases  into  an 
increased  capability  for  its  corporate  organiza¬ 
tion  [7:87] . 

Furthermore : 

There  must  be  a  balance  in  a  research  and  develop¬ 
ment  program  between  system  requirements  and  pos¬ 
sible  long-range  application  [7:87], 

Factors  which  must  be  considered  in  the  evaluation  of 
a  laboratories'  R&D  programs  are: 

1.  The  corporate  objectives; 

2.  The  mission  of  the  laboratory; 

3.  System  requirements  vs.  possible  break¬ 
throughs  ; 

4.  In-house  vs.  contractual  efforts; 

5.  Schedule  [7:89]. 

Burgess  further  analyzed  exploratory  development  programs 

over  a  five  year  period  on  the  basis  of  final  reports 
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submitted  upon  program  completion.  "To  be  more  valuable, 
papers  presented  at  symposia,  or  published  in  journals,  and 
patents  should  be  included  (7:89)."  Publications  and  pre¬ 
sentations  should  not  be  overemphasized  because  large  numbers 
of  papers  and  presentations  per  researcher  may  detract  from 
determining  laboratory  effectiveness.  The  publications  were 
categorized  in  terms  of  applicability  to  determine  how  well 
the  results  contribute  to  the  store  of  technical  knowledge. 

Note  that  the  Government  laboratory  cannot  use 
the  tools  available  to  industry,  such  as  new  product 
lines  creating  profit,  or  increased  sales,  or  both. 

The  Government  laboratory  program  culminates  in  new 
products;  however,  the  value  of  these  new  products 
must  be  assessed  in  terms  of  increased  Air  Force 
capability  rather  than  in  dollars  and  cents  [7:89]. 

Burgess  evaluated  programs  in  progress  in  relation  to  the 

following  questions  regarding  the  schedules: 

1)  How  much  will  it  cost  to  complete  the 
project? 

2)  How  much  time  will  it  take  to  complete  the 
project? 

3)  How  many  milestones  are  left  to  be  accom¬ 
plished?  [7:90]. 

Lebanoff  (39:110-122)  analyzed  the  reporting  system 
used  at  the  Air  Proving  Ground  Center  (APGC)  from  scentific 
and  engineering  personnel  to  top  management.  The  in-progress 
reporting  system  should  allow  management  to  control  the  cost, 
time,  and  quality  of  research  projects.  Project  engineers 
made  bi-weekly  reports  which  were  analyzed  using  the  tech¬ 
niques  of  frequency  tabulations,  percentage  calculations, 
weighted  ratios,  repression  line  analysis,  correlations, 
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test  of  significance,  confidence  limits,  and  importance 
ranking.  The  resultant  data  can  be  reduced  with  com¬ 
puters  to  allow  management  to  identify  the  critical 
paths  within  projects.  The  system  was  considered  to  be 
adaptable  to  the  monitoring  of  R&D  contracts  and  other 
problem  area?  such  as  the  cost,  time  consumed,  and  faci¬ 
lities  usage  in  test  projects. 

Mansfield  interviewed  35  companies  to  explore 
industrial  R&D  expenditures.  He  discussed  the  major 
topics : 


1.  What  determines  the  level  of  a  firm's 
privately  financed  research  and  development  ex¬ 
penditures? 

2.  What  is  the  relationship  between  a  firm's 
research  and  development  expenditures  and  various 
crude  measures  of  inventive  output? 

3.  Do  the  largest  firms  in  various  industries 
spend  more  on  research  and  development  relative 

to  their  size,  than  firms  one- fifth  or  one- tenth 
of  their  size  [43:21-22]? 

Mansfield  prefaced  this  exploration  by  commenting 

that  research  and  development  has  an  uncertain  outcome. 

Mansfield  states  the  following: 

For  example,  a  recent  survey  of  120  large  com¬ 
panies  doing  a  substantial  amount  of  research  and 
development  indicates  that  in  half  of  these  firms 
at  least  60  percent  of  the  research  and  develop¬ 
ment  projects  never  resulted  in  a  commercially 
used  product  or  process.  .  .  .  Moreover,  even 
when  a  project  resulted  in  a  product  that  was  used 
commercially,  the  profitability  of  its  use  was 
likely  to  be  quite  unpredictable  [43:16]. 

In  other  words,  risk  and  payoff  are  not  easily  determined 

prior  to  a  company's  decision  to  undertake  a  specific  project. 
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The  implication  is  that  an  improvement  in  the  methodology 
used  in  determining  risk  and  payoff  will  result  in  improved 
R&D  productivity. 

Mansfield  uses  data  from  several  industrial  companies 
and  mathematical  models  in  his  discussion  of  the  three  major 
topics.-  He  stated: 

1.  There  is  some  evidence  that  the  level  of  a 
firm's  research  and  development  expenditures  can 

be  reasonably  explained  well  by  a  simple  model  .... 

2.  This  model  may  be  of  use  in  forecasting  re¬ 
search  and  development  expenditures  and  in  estimating 
the  effects  on  research  and  development  expenditures 
of  changes  in  relevant  policies. 

3.  If  .  .  .  one  assumes  that  a  firm's  desired 
research  and  development  expenditures,  as  a  percent 
of  sales,  are  a  quadratic  function  of  time,  the  re¬ 
sulting  version  of  the  model  seems  quite  useful  for 
short-term  forecasting. 

4.  Except  for  the  chemical  industry,  there  is 
no  evidence  that  the  largest  firms  in  these  indus¬ 
tries  spent  more  on  research  and  development,  re¬ 
lative  to  sales,  than  did  somewhat  smaller  firms. 

In  the  petroleum,  drug,  and  glass  industries,  the 
largest  firms  spent  significantly  less;  in  the  steel 
industry  they  spent  less  but  the  difference  was  not 
statistically  significant. 

5.  Where  the  size  of  the  firm  is  held  con¬ 
stant,  the  number  of  significant  inventions  car¬ 
ried  out  by  a  firm  seems  to  be  highly  correlated 
with  the  level  of  its  research  and  development 
expenditures.  In  the  chemical  industry,  increases 
in  research  and  development  expenditures  apparently 
result  in  more  than  proportional  increases  in  in¬ 
ventive  output,  but  in  petroleum  and  steel  there 

is  no  evidence  of  either  economies  or  diseconomies 
of  scale  within  the  relevant  range.  In  most  indus¬ 
tries,  the  productivity  of  a  research  and  develop¬ 
ment  program  of  given  scale  seems  to  be  lower  in 
the  largest  firms  than  in  somewhat  smaller  firms 
[43:84-85] . 

Farris  studied  the  relationship  between  organiza¬ 
tional  factors  and  the  performance  of  151  electronics  en¬ 
gineers  "to  determine  the  extent  to  which  the  factors 
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preceded  performance  and  performance  preceded  the  factors 
(20:9),"  and  to  investigate  the  "stability  of  relationships 
and  time  lags  in  measurement  (21:87)." 

Four  factors  were  related  significantly  to 
subsequent  performance;  involvement  in  work, 
colleague  contact,  diversity  of  work  activities, 
and  number  of  subordinates .  Every  factor  studied 
(these  four,  plus  salary  and  influence  on  work 
goals)  was  related  to  previous  performance.  The 
performance- factor  sequence  was  much  more  predomi¬ 
nant  than  the  factor-performance  sequence.  An  en¬ 
gineer’s  performance  apparently  has  pervasive  con¬ 
sequences  for  his  social-psychological  working  en¬ 
vironment  [20:0]. 

The  results  of  the  Farris  self - report- questionnaire  from  re¬ 
spondents  in  1959  and  1965  found  that  relationships  between 
six  organizational  factors  (involvement  in  work,  influence 
on  work  goals,  colleague  contact,  diversity  of  work  activi¬ 
ties,  salary,  and  number  of  subordinates)  and  four  perfor¬ 
mance  measures  (contribution,  usefulness,  patents,  and  re¬ 
ports)  were  stable. 

Harrold  (29:15-23)  reported  in  1969  that  "it  is 
possible  to  predict  laboratory  performance  with  greater 
confidence  than  to  predict  the  number  of  papers  and  inven¬ 
tion  disclosures."  Fifteen  Army  laboratories  were  surveyed 
to  define  and  evaluate  measurable  performance  indicators. 

In  summary,  Harrold  (29:22-23)  found: 

1)  Generally  speaking,  knowledge  of  mea¬ 
surable  characteristics  can  help  executives  in¬ 
crease  their  efficiency  in  the  management  of 

a  military  laboratory  complex. 

2)  The  composition  of  laboratory  personnel 
is  important  to  Army  laboratory  output  and  per¬ 
formance.  Age  and  R&D  experience  of  civilian  and 
military  personnel  do  affect  laboratory  output 
and  performance. 
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3)  The  study  indicates  that  an  increase  in  R&D 
experience  for  military  personnel  of  a  laboratory 
would  add  significantly  to  its  performance  and 
output.  However,  there  are  other  considerations, 
such  as  the  military  policy  of  rotation,  that  can 
limit  the  R&D  experience  of  military  personnel. 

An  increase  in  experience  must  be  accomplished  in 
light  of  policy  and  other  unmeasurables  of  the 
military  laboratory. 

4)  While  this  study  emphasized  those  charac¬ 
teristics  that  were  measurable,  other  unmeasurables 
such  as  employee  motivation,  morale,  and  working 
environment  also  affect  laboratory  output  and  per¬ 
formance.  Administrative  policies  that  improve 
such  unmeasurables  should  continue. 

5)  Statistically,  there  was  little  relation¬ 
ship  between  the  standards  of  laboratory  output 
and  performance.  Yet,  in  each  case  large  increases 
in  obligations  ($1  million)  only  slightly  improved 
output  or  performance  of  a  single  Army  laboratory. 

6)  While  it  was  possible  to  establish  meaning¬ 
ful  input/output  and  input/performance  relationships 
future  study  over  a  period  of  years  could  consider 
"lead-lag"  elements.  That  is,  measurable  characteris¬ 
tics  of  fiscal  years  1965  and  1966  might  affect 
output  and  performance  in  subsequent  years. 

Hill  (32:10-19)  reported  observational  and  interview 
data  from  a  field  study  of  a  government  scientific  division 
(25  scientists)  to  determine  the  effect  of  organizational 
change  on  the  relationship  of  leadership  behavior  patterns 
to  productivity  and  group  morale.  Morale  was  indicated  by 
turnover  rate  and  productivity  was  measured  by  publication 
rate.  Only  those  publications  in  discipline-relevant  jour¬ 
nals  were  counted  thus  eliminating  minor  publications  and 
technical  reports  from  consideration.  Productivity  in  the 
autocratic  leadership  group  was  significantly  less  than  in 
the  group  with  democratic  leadership. 

In  a  1970  study  of  70  scientists  from  three  Air  Force 


laboratories  at  Wright-Patterson  AFB  OH,  Stahl  examined 


organization  facets  "that  would  foster  the  creativity  of 
the  scientists  or  engineers  .  .  .  (61:5)." 

A  questionnaire  methodology  testing  nine  hypotheses 
was  used  to: 


.  .  .  measure  the  degree  to  which  scientists 
and  engineers  perceive  the  organizational  factors 
to  be  present  In  their  laboratories  rather  than  to 
objectively  measure  the  degree  to  which  such  fac¬ 
tors  actually  exist  [61:6]. 

Stahl  found: 


.  .  .  that  little  differences  exist  in  the  per¬ 
ceived  presence  of  the  organizational  factors  among 
the  scientists  and  engineers  who  have  been  working 
in  the  laboratories  for  different  numbers  of  years, 
and  among  scientists  and  engineers  when  classified 
as  military  or  civilian.  Significant  differences 
exist  in  the  perceived  presence  of  certain  organi¬ 
zational  factors  among  the  scientists  and  engineers 
in  different  laboratories,  in  different  educational 
levels,  engaged  in  different  kinds  of  work,  in  dif¬ 
ferent  grades  and  among  scientists  versus  engineers 
[61:113] . 

and  these  differences  are  based  on: 

.  .  .  twenty-four  organizational  factors  that 
affect  the  creativity  of  scientists  and  engineers  in 
Air  Force  R&D  laboratories .  These  twenty-four  factors 
are : 


1.  Management  receptivity  to  new  ideas; 

2.  An  incentive  system  that  researches 
creativity ; 

3.  Time  for  independent  research; 

4.  Open  communication  channels  both  vertically 
and  horizontally  in  the  organization; 

5..  Freedom  in  scheduling  work  hours; 

6.  Freedom  in  choice  of  problem; 

7.  Freedom  and  autonomy  in  work  procedures  and 
methods ; 

8.  Recognition  for  creative  accomplishments; 

9.  Challenging  work; 

10.  Stimulating  colleagues; 

11.  Organizational  acceptance  of  conflict  and 
dissent; 

12.  Organizational  acceptance  of  nonconformity; 

13.  Employee  understanding  of  organization  objec¬ 
tives; 
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14.  Little  administrative  distractions  and 

routine  duties ; 

15.  Excellent  facilities; 

16.  Little  fear  of  failure; 

17.  Sufficient  time  to  just  think ; 

18.  Organization  structure  flexibility; 

19!  Opportunity  for  professional  society  par¬ 
ticipation; 

20.  Opportunity  to  publish; 

21.  Relaxed,  flexible  controls; 

22.  Management  encouragement  of  and  confidence 
in  employees; 

23.  Participative  management; 

24.  Dual  promotion  ladders  [61:113-114]. 

Further  breakdown  on  the  differences  are  presented  in 
Stahl’s  conclusion  (61:113-120). 

Whitley  and  Frost  (75:161-178)  states: 

Of  particular  importance  for  developing  adequate 
indexes  of  individual  and  collective  performance  is 
the  specification  of  goals.  Both  an  understanding 
of  what  goals  the  individuals  and  the  institution 
as  a  whole  are  expected  to  achieve  and  what  goals 
are,  in  fact,  seen  as  a  salient  by  the  members  of 
the  organization  are  fundamental  to  develop  use¬ 
ful  measures  of  performance  [75:162J. 

The  pros  and  cons  of  basic  and  applied  performance  measures 

for  individuals  and  organizations  in  Table  3-3  are  discussed 


(75:104)  . 


TABLE  3-3 


CLASSIFICATION  OF  PERFORMANCE  MEASURES 


OBJECTIVE 

Number  of  papers 

Number  of  books 

Number  of  citations 

Number  of  professional  awards 

Number  of  powerful  positions  held 

Individual  members  output 

Library  facilities 

Number  of  graduate  students 

Number  of  outside  visitors 

Cost  and  schedule  overruns 

Number  of  adopted  innovations 

'Parellel  projects' 


SUBJECTIVE 

Prestige  ranking  by  peers 
Prestige  ranking  by  mem¬ 
bers  of  other  groups 
Supervisor  evaluation 
Colleque  evaluation 
Executive  evaluation 
Members  evaluation  of 
organization  relative 
to  ' top  10 ’ 

University  evaluation 
Professional  society 
evaluation 


Inter-organizational  comparison  of  the  per¬ 
formance  of  individual  scientists  is  not  possible, 
we  suggest,  unless  the  objectives  of  the  research 
are  similar  in  the  organizations  concerned  [75:173]. 

The  measurement  of  performance  in  research 
laboratories,  for  example,  becomes  the  understanding 
of  how  different  solutions  are  produced  to  technical 
problems  in  different  environments.  To  improve 
the  quality  of  such  solutions ,  we  have  to  know  both 
how  they  were  reached  and  what  they  are  required  for. 
This  involves  the  detailed  and  systematic  study  of 
how  knowledge  is  created  by  scientific  activity 
which  necessitates  in  turn  investigating  what  scien¬ 
tists  do  in  identifying  and  representing  problems 
and  selecting  amd  applying  research  strategies  and 
tactics.  It  also  involves  the  study  of  the  interre¬ 
lations  between  ideas  and  products  and  processes 
both  technological  and  social.  The  problem  of 
measuring  knowledge  is  created  and  used  in  society. 
Many  of  the  difficulties  encountered  to  date  in  de¬ 
veloping  performance  measures,  we  suggest,  are  due 
to  the  failure  to  recognize  this  basic  implication 
[75:174-175] . 

Whitley  and  Frost  fully  supported  their  discussion 
of  the  objective  and  subjective  performance  measures  with 
an  extensive  bibliography  of  references  through  1971. 

Glass  (25:2-12)  summarized  the  techniques  used  by 
Doll  in  evaluating  the  effectivness  of  its  in-house  R&D 
as  of  1972. 

In  examining  Defense  R&D  laboratories ,  many 
different  types  of  appraisals  have  been  used: 
supervisory  evaluation,  program  evaluation,  spe¬ 
cial  appraisal,  visiting  committees,  and  the 
national  competition  of  laboratories  for  im¬ 
portant  programs  [25:3]. 

The  following  six  techniques  are  presented  and 
discussed: 


1.  Organizational  climate  (actions  to  im¬ 
prove  organizational  health) ; 

2.  Management  surveys  (management  improve¬ 
ment)  ; 
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3.  Organizational  model  (reorganization) ; 

4.  Model  for  failure  (emphasizes  mishandling 
pitfalls) ; 

5.  Statistical  and  economic  models  (mathema¬ 
tical  input-output  relationships); 

6.  Peer  rating  (personnel  and  organizational) . 

Participating  management,  including  the  development 

concept  paper  (DCP),  was  experimentally  being  applied  to  seg 

ments  of  DoD  research  and  technology  programs. 

Instead  of  trying  to  relate  organizational  effective¬ 
ness  to  a  generalized  model  or  set  of  criteria, 
this  concept  permits  the  appraisal  of  an  organiza¬ 
tion  based  upon  tailored  goals  that  are  specific  to 
that  particular  organization.  In  the  long  run,  this 
is  considered  to  be  the  most  useful  and  meaningful 
approach  to  laboratory  evaluation.  The  trick  will 
be  to  define  the  goals  and  objectives  in  quantitative 
terms  and  to  develop  related  tools  and  performance 
measures  [25:11]. 

Vincent  and  Mirakhor  (72:45-53)  analyzed  the  ques¬ 
tionnaire  responses  from  94  scientists  and  engineers  in  the 
Missile  Systems  Laboratory  of  the  U.S.  Army  Missile  Cammand 
This  empirical  investigation  revealed  that: 

1.  Job  satisfaction  is  a  multidimensional 
factor; 

2.  The  results  generally  agree  with  similar 
study  results; 

3.  There  is  a  statistically  significant  re¬ 
lationship  between  productivity  and  satisfaction 
(72:45). 

Productivity  was  objectively  measured  by: 

1.  The  number  of  professional  journal  publi¬ 
cations  ; 

2.  The  number  of  patents  or  patent  applications; 

3.  The  number  of  manuscripts  or  unpublished 
technical  reports  (72:46). 
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Considering  satisfaction,  those  with  high  satisfac¬ 
tion  publish  at  a  higher  rate,  at  a  lower  age  and  taper 
off  less  sharply.  Those  of  low  satisfaction  publish 
later  and  taper  off  sharply. 

It  should  be  noted  that  the  most  productive  group 
with  respect  to  publication  of  papers  is  the  group 
with  high  job  satisfaction  in  the  10-25  thousand 
dollar-per-year  pay  range.  These  persons  were  also 
predominantly  in  the  26-to-35  year  age  group.  Further¬ 
more,  the  number  of  papers  published  is  only  an  indi¬ 
cator  of  productivity  and  does  not  give  consideration 
to  the  all-important  matter  of  quality  of  the  paper. 

Patent  disclosures  or  inventions  are  considered 
to  be  the  most  important  and  meaningful  productivity 
indicator  of  a  scientist  or  engineer  working  in  a 
laboratory  [72:48]. 

The  highest  rate  of  patent  disclosures  occurs  with  the  30-35 
age  group  with  high  satisfaction  who  is  in  the  15-19  thousand 
dollar  pay  range  (72:48).  The  presentations  indicator  of 
productivity  reflects  that  the  40-45  age  group  in  the  20-25 
thousand  dollar  pay  range  with  25  years  experience  give  more 
presentations.  The  net  result  is  that  young  low-salaried 
researchers  produce  more  publications  and  patents,  and  middle 
age  researchers  tend  to  give  more  presentations  (transmission 
of  ideas  and  experience)  (72:50). 

Andrews  and  Farris  (3:185-200)  derived  five-year 
panel  data  from  approximately  100  scientists  in  a  NASA  lab¬ 
oratory  to  determine  the  effects  of  time  pressure  and  in¬ 
dividual  performance.  Respondents  completed  a  lengthy  ques¬ 
tionnaire  in  both  1965  and  in  1970.  Individual  performance 
was  evaluated  by  selected  laboratory  professionals.  Inno¬ 
vation,  productiveness,  and  usefulness  were  the  performance 
criteria  utilized  to  express  how  well  or  poorly  each 

i  ! 
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researcher  performed  relative  to  other  researcher's  simi¬ 
lar  training  and  experience.  They  are  defined  as  follows: 

The  performance  criteria  included  the  following: 

Innovation — the  extent  the  man's  work  had 
"increased  knowledge  in  his  field  through  lines  of 
research  or  development  which  were  useful  and  new." 

Productiveness — the  extent  the  man's  work  had 
"increased  knowledge  along  established  lines  of 
research  or  development  or  as  extensions  or  re¬ 
finements  of  previous  lines,"  and 

Usefulness — the  extent  the  man's  work  had  been 
"useful  or  valuable  in  helping  his  R&D  organiza¬ 
tion  carry  out  its  responsibilities"  [3:188]. 

Andrews  and  Farris  conclude! that  the  highest  per¬ 
forming  researchers  tended  to  desire  relatively  large 
amounts  of  pressure,  and  that  performance  was  likely  to 
suffer  when  pressure  differed  substantially  from  the  de¬ 
sired  pressure. 

Edwards  and  McCarry  (18:34-41)  reported  in  a  1973 
literature  review  that  considerably  more  study  is  needed  to 
accurately  evaluate  the  quality  and  quantity  of  the  work 
of  a  research  staff.  The  definition  of  scientific  output 
is  given  as  follows: 

1.  The  discovery  of  new  facts; 

2.  The  invention  of  new  methods  of  doing  things; 

3.  Combining  known  concepts  to  create  new 
devices  [18:34]. 

Overall  performance  methods  reviewed  are  : 

1.  ratings/rankings; 

2.  number  of  patent  disclosures; 

3.  contributions  to  scientific  knowledge; 

4.  contribution  to  the  organization 
(18:34-35) . 

The  quantity  of  written  output  reviewed  consists  of: 

1.  number  of  papers  published; 

2.  number  of  citations; 

3.  number  of  books  published; 
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4.  number  of  patents; 

5.  number  of  unpublished  reports; 

6.  number  of  patent  applications; 

7.  number  of  papers  presented  at  pro¬ 
fessional  meetings  (18:35-36). 

The  quality  of  output  reviewed  consists  of: 

1.  ratings/rankings; 

2.  significance  of  research  reports; 

3.  relevance  -  timeliness  of  research 
reports ; 

4.  organization  -  lucidity  of  research 
reports ; 

5.  originality  of  research  reports; 

6.  elegance  -  accuracy  -  exhaustiveness 
of  research  reports ; 

7.  complexity  of  the  research  assignment; 

8.  amount  of  research  supervision  received; 

9.  demands  of  originality  and  lack  of 
guidelines ; 

10.  scientific  qualifications  and  contribu¬ 
tions  of  the  scientist; 

11.  citation  count  (Science  Citation  Index) 
(18:36-38) . 

The  summary  states: 

A  review  of  the  studies  on  measurement  of  scien¬ 
tific  performance  reveals  very  little  agreement  from 
one  investigator  to  another  as  to  what  constitutes 
scientific  output  or  what  measures  should  be  used 
to  reflect  the  output.  ...  A  few  in-depth  studies 
of  scientific  performance  have  revealed  that  scien¬ 
tific  output  is  multidimensional  and  cannot  be  satis¬ 
factorily  measured  by  any  one  criterion  alone.  .  .  . 

A  factor  analytic  study  which  reduces  scientific 
performance  to  a  few  practical,  workable  components 
would  have  great  utility  for  the  research  community 
[18:40]. 

McCarry  and  Edwards  (45:  *3-  '9)  also  interviewed 

72  research  biologists  in  four  Janadian  Government  labora¬ 
tories  in  1973  to  determine  the  relationship  between  scien¬ 
tific  accomplishment  and  the  characteristics  of  effective 
organizational  climates.  The  following  nine  performance 
measures  were  utilized: 
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1.  RanJc  (by  peers)  of  the  incumbent's  total 
bench  work  "mission-oriented"  productivity; 

2.  Rank  (by  peers)  of  the  incumbent's 
creativity  of  role  performance,  i.e.,  originality, 
stimulation,  and  uniqueness  of  mission-related 
problem  solving; 

3.  Departmental  percentile  standing; 

4 .  Extradepartmental  recognition  score , 
i.e.,  invited  addresses,  journal  reader  or 
editorship,  etc.; 

5.  Rank  (by  peers)  of  the  incumbent's  com¬ 
munications  performance,  i.e.,  clarity  and  organi¬ 
zation  in  receiving,  comprehending  and  transmitting 
range  of  oral  and  written  contents  (it  was  neces¬ 
sary  to  partial  out  effects  of  friendship  stuc- 
ture  on  communications  rank) ; 

6.  Quality; 

7.  Originality  ratings  of  refereed  journal 
publications  by  discipline  peers; 

8.  Citation  rate,  viz.,  number  of  citations 
received  per  paper  over  a  five  year  time  interval; 

9.  Publication  rate,  weighted  for  differential 
authorship,  viz.,  number  of  papers  per  year  with  a 
correction  for  author  position  [45:441-442]. 

O'Brien  assumed  effectiveness  is  a  function  of  the 
quantity  and  quality  of  the  output  of  the  General  Systems 
Engineering/Technical  Director  (GSE/TD)  process  and  that 
effectiveness  is  measured  in  terms  of  documented  information 
(51:7).  The  paper  reports  a  study  of  812  scientists  in  23 
engineering  teams  in  the  organization  of  the  Deputy  for 
Engineering,  Aeronautical  Systems  Division  in  1973.  This 
one-month  study  documented  all  their  output  quantitatively 
and  qualitatively.  All  written  and  verbal  information 
being  transmitted  outside  the  GSE/TD  team  was  recorded  and 
assessed  as  team  output.  Output  quality  was  quantified 
with  weightings  in  a  "sum  approach"  and  a  "product 
approach."  Output  quantity  was  determined  from  the  number 
◦f  pages  of  documents  and  verbal  communications  memoranda 
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in  the  2037  output  assessment  forms  completed  and  processed. 

Harrison  (30:234-241)  reported  the  results  of  an 
empirical  study  of  95  scientists  in  three  large  research  lab¬ 
oratories  to  relate  perceived  role  performance  to  the  sys¬ 
tem  of  management.  The  four  perceived  performance  measures 
were  as  follows: 

1.  Esteem  of  fellow  scientists; 

2.  Contribution  to  knowledge  in  the 
field; 

3.  Contribution  to  management  objec¬ 
tives  ; 

4.  Sense  of  personal  achievement. 

Harrison  concluded  with  the  following  statements: 

.  .  .  the  hypothesis  that  the  more  organic 
the  system  of  management  the  higher  the  perceived 
role  performance  of  the  individual  scientist  was 
accepted. 

The  evidence  suggests  that  perceived  and,  pre¬ 
sumably,  actual  role  performance  in  a  research  lab¬ 
oratory  will  be  improved  if  the  scientist  is  en¬ 
couraged  and  permitted  to  participate  actively  and 
regularly  in  the  setting  of  objectives  and  making 
of  decisions  [30.241]. 

Katzen  (35:24-28)  graphically  presented  the  perfor¬ 
mance  profiles  of  11  scientists  in  the  form  of  ratings  ver¬ 
sus  a  seven  year  time  frame. 

.  .  .  engineers  are  involved  in  a  wide  variety 
of  activities,  ranging  from  fundamental  research 
through  applied  research  and  development  to  design, 
operation,  plant  and  technical  services,  sales,  etc. 

The  output  for  each  type  of  function  is  different, 
which  again  is  difficult  to  relate  by  quantitative 
measurement.  In  certain  activities  such  as  re¬ 
search,  development,  and  design,  the  quality  of 
the  output  is  even  more  important  than  the  quanti¬ 
ty,  and  ♦■he  innovative  or  inventive  value  is  a 
major  factor  which  does  not  lend  itself  to  quan¬ 
titative  measurement  [35:24]. 
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Katzen’s  technique  for  performance  measurement 
consists  of  an  effectiveness  measure  and  an  efficiency 


measure  expressed  in  percentages.  Effectiveness  was  mea¬ 
sured  as  a  percentage  of  working  hours  the  engineer  has 
put  in  on  specific  projects  over  the  total  hours  monthly 
available  (excluding  unassigned  time,  vacation  time,  sick 
leave,  and  meetings).  Efficiency  was  determined  by  assessing 
whether  or  not  all  the  effective  work  hours  were  fully  pro¬ 
ductive.  The  ratings  in  Table  3-4  were  applied  to  11 
scientists  (35:26). 


TABLE  3-4 

EFFECTIVENESS  AND  EFFICIENCY  RATINGS 
EFFECTIVENESS  EFFICIENCY 


90%  -  highly  effective 

85%  -  very  satisfactory 

80%  -  good,  but  subject  o 
improvement 

75%  -  fair  to  poor,  inex¬ 
perienced 


95%  -  excellent 
90%  -  very  good 
85%  -  good 
80%  -  fair 

75%  -  poor,  inexperienced 


Ferguson  (23:33-35)  reported  the  implementation  of 
an  MBO  program  at  Erling  Riis  Laboratory  to  develop  and 
carry  out  research  projects.  Key  Work  Objections  (KWO) 
define  priorities  and  establish  activities  and  are  a  main 
input  for  annual  performance  appraisals. 
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Being  involved  in  setting  goals,  the  researcher 
is  more  apt  to  be  working  in  his  true  areas  of 
interest.  .  .  .  Productivity  is  more  easily  mea¬ 
sured  by  the  key  work  objectives  system  and  per¬ 
formance  appraisals  are  more  objective.  In  general, 
we  have  found  that  the  use  of  KWQ's  stimulate  team¬ 
work,  orient  effort,  and  establish  definite,  objec¬ 
tive  goals  that  lead  to  increased  productivity 
[23:34] . 

Whelan  (74:14)  reported  in  1976  that  Union  Carbide 
employs  a  one  sheet  questionnaire  form  to  descriptively 
categorize  projects  and  measure  R&D  effectiveness.  The 
R&D  categorization  questionnaire  informs  corporate  manage¬ 
ment  of  the  following: 

1.  Documentable  estimates  of  the  relative 
amounts  of  corporate  R&D  effort  being  devoted  to 
maintenace  of  existing  business  versus  the  quest 
for  new  business; 

2.  Relative  effort  on  high-risk  versus  low-risk 
objectives ; 

3.  Other  parameters  of  the  total  R&D  program. 

With  the  availability  of  data  for  several  suc¬ 
cessive  years  obtained  on  a  consistent  basis,  it 
is  possible  to  see  trends  in  R&D  posture  reflect¬ 
ing  the  R&D  response  to  dynamic  corporate  goals 
as  well  as  the  impact  of  budget  variations  on 
R&D  objectives  [74:14]. 

Koser  surveyed  135  scientists/engineers  in  an  Air 
Force  laboratory  in  a  1976  thesis  study  of  the  relationships 
between  selected  organizational  and  individual  variables 
and  an  objective,  quantifiable  measure  of  scientist/engineer 
productivity.  The  study  includes  "...  the  selection  of 
variables  that,  in  previous  studies,  have  been  shown  to  be 
correlated  to  some  definition  of  scientific  and  engineering 
performance  (38:3)."  Koser  utilized  a  set  of  variables 
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similar  to  the  set  investigated  by  Stahl  for  the  purpose 
of  comparison.  Koser  briefly  reviews  the  publications  of 
Stahl,  Vincent  and  Mirakhor,  Pelz  and  Andrews,  Mullins, 
Meltzer  and  Salter,  Eidusun,  Hill,  Farris,  Andrews  and 
Farris,  Pelz,  and  Martino.  The  final  list  of  output 
consisted  of  : 

1.  technical  reports; 

2.  work  unit  planning  documents; 

3.  journal  articles; 

4 .  procurement  packages ; 

а.  evaluation  of  proposals; 

б.  computer  codes; 

7.  technical  presentations; 

8.  management  presentations  [38:18). 

Stahl  and  Koser  (63:20-24)  used  these  same  eight  output  vari¬ 
ables  in  a  1978  publication. 

Stevens  (66:9-10)  used  the  terms  innovation,  ori¬ 
ginal,  useful,  and  productivity  (as  defined  by  Stahl 
(62:91-92))  to  investigate  rewards  in  Air  Force  R&D.  Stevens 
used  the  following  quantifiable  output  variables: 

1.  New  or  improved  products; 

2.  New  or  improved  process  techniques; 

3.  Patents/patent  applications; 

4.  Published  papers  in  technical  or  pro¬ 
fessional  journals; 

5.  Air  Force  technical  reports; 

6.  Air  Force  technical  memorandums; 

7.  Books; 

8.  Manuscripts; 

9.  Hardware/software  specifications; 

10.  Test  plans; 

11.  Test  reports; 

12.  Statements  of  work; 

13.  Requests  for  proposal; 

14.  Oral  presentations  to  technical/pro¬ 
fessional  audiences  [66:35]. 

The  survey  involved  278  scientists/engineers  in  one  Air 

Force  laboratory  to  determine  relationships  between 
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productivity  and  rewards  for  output.  A  discussion  of 
results  is  provided  in  Stevens'  thesis  (66:101-108). 

Stahl  (62:33-74)  summarized  empirical  studies 
between  1951  and  1974  relating: 

.  .  .  specific  organizational  variables  with  a 
measure  of  output  of  scientists  and  engineers  in  an 
R&D  setting,  or  with  a  measure  of  original  output 
in  a  controlled  laboratory  experimental  setting 
[62:33]. 

The  1976  Stahl  study  (62:93-188)  analyzed  41  organizational 
predictor  variables  and  the  two  criterion  variables  pro¬ 
ductivity  and  innovation  defined  as  follows: 

1.  Productivity:  Quantity  of  scientific 
output. 

2.  Innovation:  Quality  of  scientific 
output,  original  and  useful  [62:91]. 

Three  Air  Force  laboratories  (154  civilian  and  military 

scientists/engineers)  were  surveyed.  The  peer  ratings  for 

innovation  and  productivity  within  the  same  work  group 

were  found  to  be  reliable  criterion  measures.  Productivity 

(quantity)  reliability  was  found  to  be  consistently  just 

higher  than  the  innovation  (quality)  reliability.  "This 

is  somewhat  expected  since  productivity  might  have  been 

an  easier  construct  to  grasp  than  innovation  (62:122)." 

Stahl  also  found: 

A  significant  positive  relationship  between 
the  productivity  ratings  and  number  of  Air  Force 
Technical  Reports  and  Technical  Memorandums. 

A  strong  positive  relationship  between  the 
innovation  and  productivity  scores.  This  ob¬ 
served  relationship  agrees  with  three  other 
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studies  cited  in  the  literature  that 
also  found  strong  positive  relationships 
between  the  two  criteria  [62:180]. 

For  the  given  definitions,  productivity  and  inno¬ 
vation  measures  were  shown  to  be  reliable  and  valid  and 
have  utility  in  measuring  the  productivity  and  innovation  of 
scientists/engineers  in  R&O  (62:81).  The  most  signifi¬ 
cant  findings  are  as  follows  (62:181-184): 

1.  Rewards  for  innovation  are  positively 
related  to  innovation,  and  to  productivity,  and 
to  the  joint  criteria  at  both  the  individual  and 
group  levels,  and  is  the  variable  most  consistently 
related  to  performance. 

2.  Communication  with  other  professionals 
within  the  group  on  technical  matters  is  consis¬ 
tently  related  to  performance.  It  is  positively 
related  to  innovation,  productivity,  and  the 
criteria  at  the  group  level. 

3.  The  group  leader's  empathy  and  partici¬ 
pation  on  goal  setting  are  both  positively  re¬ 
lated  to  productivity  and  to  the  joint  criteria 

at  both  the  individual  and  group  level,  but  neither 
variable  is  related  to  innovation. 

4 .  The  age-education  variables :  The  age  x 
education  interaction  variable,  age,  length  of 
scientif ic/engineering  experience,  length  of 
federal  government  employment  as  a  scienti¬ 
fic/engineering  experience,  length  of  federal 
government  employment  as  a  scientist/ engineer 
and  education  variables  are  all  consistently 
related  to  performance  at  the  individual  level. 

All  are  negatively  related  to  innovation,  and 
to  productivity,  and  to  the  joint  criteria. 

The  age  x  education  interaction  variable  has 
the  strongest  negative  association  with  innova¬ 
tion. 

5.  Autonomy  is  not  found  to  be  related  to 
either  innovation  or  productivity. 

6.  Communication  with  the  group  leader  on 
technical  matters  is  curiously  negatively  re¬ 
lated  to  innovation  in  both  the  individual  and 
group  level  regression  analysis. 

Stahl  and  Steger  published  two  similar  papers  in 

1977  (64:71-76;  65:35-38)  in  the  open  literature  revealing 
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much  the  same  information  as  previously  given  by  Stahl 


(62:180-184).  But  the  studies  also  revealed: 

.  .  .  that  those  individuals  classified  as 
highly  productive  had  an  average  3.2  publications 
for  every  year  of  professional  work,  the  innovative 
group  had  an  average  of  2.5  publications  per  year, 
and  those  classified  as  non- innovative  and  nonpro¬ 
ductive  had  an  average  publication  count  of  .18 
per  year  of  professional  activity  [64:37], 

Communication  with  other  professionals  within 
the  group  on  technical  matters  was  consistently  re¬ 
lated  to  performance.  It  was  positively  related 
to  innovation,  productivity  and  the  bivariate 
criteria  at  the  individual  level,  as  well  as  to 
innovation  at  the  group  level.  .  .  . 

Participation  on  goal  setting  and  the 
group  leader's  empathy  were  positively  related 
to  productivity  and  to  the  joint  criteria  at 
both  the  individual  and  group  level.  Neither 
variable  is  related  to  innovation  [64:74-75]. 


Bakalkin  and  Pita k  (5:1—3)  reported  that  over  a  five- 
year  period  in  the  Ukranian  Scientific  Research  Institute 
of  Refractories, 

.  .  .  184  projects  were  undertaken,  from 
them  133  were  introduced  into  industry  and  49 
were  passed  on  for  use  in  planning  and  design. 

The  actual  cost  benefit  per  ruble  of  expen¬ 
ditures  grew  from  1.57  rubles  in  1971  to  3  rubles 
in  1975.  .  .  .  The  output  per  worker  (scientific 
research  and  junior  technical  personnel)  during 
this  period  increased  from  4,500  rubles  to  5,080 
rubles  [5.2] . 

Thus,  technology  transitions  (to  industrialization)  and  cost 
benefits  per  ruble  expenditure  were  two  Soviet  applied  sci¬ 
ence  performance  measures  in  1977. 

Corbin  (12:1-35)  explored  the  relationships  of 
individual  and  supervisory  variables  to  the  productivity 
(quantity  of  output)  and  job  satisfaction  (Hoppock)  of 


72 


326  surveyed  scientists  and  engineers  in  the  Air  Force 
Flight  Dynamics  Laboratory  ( AFFDL) .  Only  quantitative 
output  measures  similar  to  those  used  at  AFFDL  by  Stevens 
(66:35)  were  used.  A  complete  statistical  analysis  re¬ 
vealed  the  following: 

No  relationship  was  found  between  produc¬ 
tivity  and  job  satisfaction.  Although  higher 
education,  grade,  and  experience  were  associated 
with  higher  productivity,  no  single  predictor 
variable  was  shown  to  be  significantly  associated 
with  all  six  of  the  productivity  variables.  .  . 

(12:124] . 

Coile  tested  the  square  root  theory  of  scientific 
publication  productivity:  the  postulate  model  stating  that 
"half  of  all  scientific  papers  in  a  field  are  contributed 
by  a  few  highly  productive  authors  numbering  approxi¬ 
mately  the  square  root  of  the  total  of  scientific  authors 
(8:1)."  The  scientific  productivity  of  chemists  listed 
in  Chemical  Abstracts  whose  names  began  with  the  letter 
A  or  B  were  selected  -  a  total  of  6,891. 

The  84  most  productive  chemists,  the  square 
root  of  6,891,  contributed  4,207  papers  of  the 
total  22,839  papers,  or  18  percent — considerably 
less  than  the  50  percent  asserted  by  the  model 
[8:11. 

Porter  (54:28-30)  described  the  following  five  bene¬ 
fits  Mobil  Research  has  experienced  since  I960  from  es¬ 
timating  the  value  of  the  results  of  completed  R&D; 

1.  Motivate  researchers? 

2.  Help  identify  high  business  risk 
objectives  ? 

3.  Demonstrate  research  productivity; 

4.  Identify  productive  research  areas; 

5.  Increase  confidence  in  predictive 
evaluations. 
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In  support  of  item  #3,  demonstrate  research  productivity. 
Porter  states  the  following: 

The  post  audits  show  that,  on  the  average, 
the  value  of  our  research  exceeds  its  cost 
several  times.  And  they  have  shown  corporate 
management  that  the  quantifiable  value  of  re¬ 
search  fluctuates  widely  from  year  to  year, 
because  most  of  the  value  is  derived  from  a 
few  very  large  accomplishments,  and  in  some 
years  no  large  ones  mature  (54:29) . 

In  1978,  Stahl  and  Koser  (63:20-24)  used  the  eight 
output  variables  identified  by  Koser  (38:18)  and  found: 

The  net  result  of  using  the  weighting  metho¬ 
dology  was  a  set  of  highly  reliable  weights  which 
implies  that  it  was  a  viable  methodology.  How¬ 
ever,  its  very  high  correlation  with  unweighted 
output  implies  that  it  is  essentially  the  same 
measure.  Due  to  the  restriction  of  range  arising 
from  the  two  year  time  limitation  on  output,  it  is 
suggested  that  such  a  weighting  scheme  be  used 
only  for  longer  time  periods  of  output  [63:24]. 

A  1978  technical  report  by  Stahl  (a  follow-on 
analysis  of  Koser's  1976  thesis)  (60:1-15)  again  used  the 
eight  output  variables  identified  by  Koser  (38:18)  to 
investigate  the  measurement  and  prediction  of  produc¬ 
tivity  in  the  Air  Force  Weapons  Laboratory.  Supervisors 
and  scientists/engineers  ranked  the  outputs  similarly 
except  for  technical  reports. 

The  producers,  on  the  average,  are  more  edu¬ 
cated,  communicate  with  other  scientists  and 
engineers  within  their  own  section  more  fre¬ 
quently,  attend  more  professional  society 
meetings,  are  concentrated  in  greater  num¬ 
bers  in  one  office,  and  have  a  stronger  per¬ 
ception  of  being  rewarded  for  productivity 
than  their  nonproductive  counterparts  [60.13]. 


74 


The  Hughes  Aircraft  Company  published  the  results 


of  Phase  I  and  Phase  II  studies  of  R&D  productivity  im¬ 
provement  in  1978  covering  the  following  main  topics: 

1.  Productivity  improvement  considerations; 

2.  Improving  operational  productivity; 

3.  Improving  managerial  productivity; 

4.  Improving  employee  productivity; 

5.  Productivity  profiles  [33:183], 

The  bibliography  (33:107-175)  is  divided  into  the 
same  categories  and  represents  the  largest  published  sin¬ 
gle  source  reference  for  R&D  productivity  publications 
encountered  by  the  investigator. 

The  initial  study  phase  (1973-74)  involved 
(1)  participation  of  27  organizations  in  indus¬ 
try  (primarily  aerospace) ,  government,  and  educa¬ 
tion,  (2)  a  survey  of  350  R&D  managers  regarding 
currently  used  techniques  for  evaluating  and  im¬ 
proving  productivity,  (3)  the  services  of  13  promi¬ 
nent  consultants,  (4)  attendance  at  eight  produc¬ 
tivity  seminars,  and  (5)  an  extensive  literature 
search.  ,  .  . 

A  second  study  phase  (1975-77)  involved  (1) 
participation  of  an  additional  32  organizations 
in  industry  (primarily  consumer  product) ,  govern¬ 
ment,  and  education,  (2)  a  survey  of  2000  R&D 
managers  and  senior  technical  personnel  to 
identify  counter  productive  factors  and  effec¬ 
tive  techniques  for  dealing  with  these  factors, 

(3)  the  services  of  an  additional  15  prominent 
consultants,  (4)  attendance  at  15  productivity 
seminars,  and  (5)  a  continuation  of  the  litera¬ 
ture  search.  .  .  . 

Study  findings  clearly  indicate  that  skilled, 
responsible  management  and  superior  productivity 
are  inseparable.  .  .  . 

The  study  findings  also  show  that  the  speci¬ 
fic  approach  to  improving  productivity  is  unique 
to  each  individual  and  organization;  there  is  no 
universal  formula.  However,  there  are  certain 
basic  orinciDles  which  encomoass  oroductivity 
improvement  and  generally  apply  to  all  organiza¬ 
tions  [33: vi- vii ] . 
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The  key  factors  to  productivity  are  given  as  effec¬ 
tiveness,  efficiency,  and  value.  The  section  titled  evalu¬ 
ating  R&D  productivity  (33:39-47)  was  of  primary  interest. 

The  Hughes’  study  participants  generally  agreed  that: 

1.  Management  should  periodically  evaluate 
R&D  productivity. 

2.  Line  managers  best  evaluate  their  own  or¬ 
ganizations. 

3.  Frequent  informal  assessment  is  desirable. 

4.  Group  output  can  typically  be  evaluated  in 
terms  of : 

a.  actual  design  performance  versus  require¬ 
ments  ; 

b.  actual  cost/schedule  versus  planned 
cost/schedule; 

c.  overall  customer  satisfaction. 

5.  Evaluation  must  be  tailored  to  each  application. 

6.  Useful  evaluation  techniques  include: 

a.  work  sampling; 

b.  productivity  ratios  and  trends; 

c.  patterns  of  performance. 

7.  Any  attempt  to  evaluate  productivity  should 
consider  employee  response  to  that  evaluation. 

8.  The  evaluation  of  group  output  is  psycho¬ 
logically  more  acceptable  than  individual  evaluation 
[33:39-47] . 

Decotiis  and  Dyer  (15:17-22)  interviewed  twenty  in¬ 
dustry  R&D  scientists  and  engineers  to  identify  the  following 
five  dimensions  of  project  performance: 

1.  Manufacturability  and  business  performance; 

2.  Technical  performance; 

3.  Efficiency; 

4.  Personal  growth  experience; 

5.  Technological  innovativeness. 

The  interview  also  empirically  identified  the  follow¬ 
ing  12  determinants  of  project  performance: 

1.  Management  support; 

2.  Inter-organizational  relations; 

3.  Sponsor  relations; 

4.  Transfer  management; 

5.  Planning  and  stability  of  specifications 
and  designs; 
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6.  Project  leader-functional  manager  relations; 

7.  Clarity  of  project  leader  role; 

8.  Project  members'  skills  and  cooperation; 

9.  Communication,  decision-making,  and  personnel 
utilization; 

10.  Planning  and  scheduling; 

11.  Control  procedures; 

12.  Leadership  [15:18]. 

DeCotiis  and  Dyer  suggested  a  model  to  predict  project  per¬ 
formance  based  on  the  12  determinants  above. 

Dekok  (16:75)  and  Rutley  (57:48)  used  the  same 
list  of  equally  weighted  productivity  categories  in  their  as 
sessments  of  team  development  at  the  Air  Force  Flight 
Dynamics  Laboratory  (AFFDL) .  The  list  consists  of  the 
following: 


1.  Published  papers  in  professional  or  technical 
journals; 

2.  Technical  reports; 

3.  Technical  memorandums  or  test  data  reports; 

4.  Presentations  at  symposia,  meetings  of  pro¬ 
fessional  organizations,  and  technical  conferences; 

5.  Hardward/sof tware  specifications,  statements 
of  work,  requests  for  proposals; 

6 .  In-house  studies ,  technical  and/or  managerial 
assessments ; 

7.  Presentations  to  general  officer-level  audi¬ 
ences; 

8.  Professional  or  technical  committee  partici¬ 
pation  (external  to  laboratory) . 

The  National  Research  Council  (NRC)  (48:50-87)  re¬ 
viewed  the  official  government  measures  of  productivity 
(concepts,  methods,  and  sources). 

Following  a  brief  overview  of  government 
measures,  the  rest  of  this  chapter  is  divided 
into  sections  on  each  of  the  major  classifi¬ 
cations  of  the  economy  for  which  official  pro¬ 
ductivity  measures  are  prepared:  economy-wide, 
industry  divisions,  selected  detailed  industries, 
and  the  federal  government  [48:50]. 
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Output  indexes  for  detailed  industries  and  selected  federal 
government  activities  are  given  (48:66-78)  and  referenced 
here  as  typical  non  R&D  measures  of  productivity.  Some  of 
these  indexes  are  as  follows: 

1.  Unweighted  quantity  index; 

2.  Unit- value  weighted  quantity  index; 

3.  Unit-employee-hour  weighted  quantity  index. 

NRC  also  reported  the  shortcomings  and  limitations  of  output 
and  input  measures. 

Day  (13:61-66;  14:55-60)  interviewed  top  executives 
in  a  two-part  article  directed  toward  solving  the  mystery  of 
productivity  measurement.  Day  quoted  Rockwell  vice  presi¬ 
dent  of  operations  Edward  Loesur  as  stating: 

"Many  engineering  and  office  managers  abhor 
the  thought  of  measuring  their  outputs.  .  .  . 

Yet  engineering  managers  will  readily  talk  about 
performance  improvements  in  terms  of  weight  re¬ 
moved,  reliability  attained,  or  new  performance 
features  of  products  they  have  designed."  [13.65], 

Universal  productivity  measures  may  never  be  developed,  but: 

Edward  J.  Freney  Associates,  a  Redding, 

Connecticut,  human  resources  consulting  firm, 
claims  that  performance  standards  can  be  es¬ 
tablished  for  any  job — even  creative  ones. 

The  key  factor  is  using  the  workers  to  help 
establish  them.  Once  they're  agreed  upon, 
employees  will  measure  their  own  output  and 
quality,  and  prompt  and  consistent  feedback 
encourages  improvement  [13:66]. 


Discussion 


The  literature  review  consists  of  13  theoretical 
publications  and  41  empirical  publications  chronologically 
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arranged  and  presented  from  1959  to  1980.  What  does  this 
computation  tell  us?  Is  there  agreement  between  the  theo¬ 
retical  and  empirical  literature  or  do  the  two  sets  of 
publications  represent  conflicting  data  and  views?  Does 
the  empirical  data  support  or  deny  the  theoretical  lines 
of  thought?  This  section  examines  the  consistencies,  con¬ 
tradictions,  and  novelities  (as  identified  in  the  litera¬ 
ture  review)  concerning  the  following  productivity  measure¬ 
ment  categories: 

1.  Objective  criteria; 

2.  Subjective  criteria; 

3.  Both  objective  and  subjective 
criteria; 

4.  Situation  specific. 

Consistencies .  Tables  3-5  and  3-6  reveal  the  percentage  of 
theoretical  and  empirical  publications  which  use  productivity 
measurement  indicators  which  are  subjective  only,  objective 
only,  or  both  subjective  and  objective.  The  theoretical 
literature  (Table  3-5)  is  applicable  to  both  industry  and 
government  and  therefore  results  in  a  single  row  of  "combined 
government  and  industry"  data.  The  empirical  literature 
(Table  3-6)  "combined  government  and  industry"  row  is  a  sum¬ 
mation  of  the  singularly  identified  government  and  industry 
data.  Four  empirical  publications  are  not.  included  in 
Table  3-6  because  they  were  literature  reviews.  Empirical 
percentages  for  government  and  industry  using  both  subjective 
and  objective  productivity  indicators  are  very  similar  (58% 
and  62%  respectively)  and  are  collectively  higher  than  the 

theoretical  value  for  "combined  government  and  industry" 
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TABLE  3-5 


THEORETICAL  LITERATURE  STATISTICS 


SUBJECTIVE  ONLY 

OBJECTIVE 

ONLY 

BOTH 

TOTAL 

Combined 

23% 

31% 

46% 

government 

industry 

and  3 

4 

6 

13 

TABLE  3 

-6 

EMPIRICAL  LITERATURE  STATISTICS 

SUBJECTIVE  ONLY 

OBJECTIVE 

ONLY 

BOTH 

TOTAL 

Government 

8% 

33% 

58% 

2 

8 

14 

24 

Industry 

23% 

15% 

62% 

13 

3 

2 

8 

Combined 

14% 

27% 

59% 

government 

industry 

and  5 

10 

22 

37 

(46%).  The 

empirical,  objective 

only  government 

value 

of 

33%  more  than  doubles  that  of  industry  (15%) .  Both  the 
government  and  the  combined  government  and  industry  percen¬ 
tages  (33%  and  27%  respectively)  roughly  equal  the  theoreti¬ 
cal,  objective  only  value  of  31%.  The  empirical,  subjective 
only  government  value  is  only  8%  whereas  the  empirical 
subjective  only  industry  value  is  23%  (equal  to  the  theore¬ 
tical,  subjective  only  combined  value).  In  summary,  these 
comparisons  indicate  that  (for  the  literature  review) : 

1.  Both  government  and  industry  utilize  a  com¬ 
bination  of  subjective  and  objective  productivity 


indicators  (59%)  more  than  the  theory  suggests 
should  be  done  (roughly  22%  more) . 


2.  Government  laboratories  rely  on  objective 
only  indicators  to  the  same  degree  that  the  theoret¬ 
ical  literature  suggests  they  should,  whereas  in¬ 
dustry  laboratories  appear  not  to  be  influenced  by 
the  theoretical  findings. 

3.  Industry  laboratories  rely  on  subjective 
only  indicators  to  the  same  degree  that  theoretical 
publications  suggest  they  should,  whereas,  government 
laboratories  do  not  follow  the  suggestions  of  the 
theoretical  literature. 


TABLE  3-7 

CONSISTENTLY  INDENTIFIED  R&D  PRODUCTIVITY  INDICATORS 
(OBJECTIVE  AND  SUBJECTIVE) 


Budget  (costs,  cost  ratios,  cash  flow) 

Checklist 

Competitive  comparisons  (rankings,  ratings) 

Customer  satisfaction 

Effectiveness  (objectives  accomplished) 

Efficiency  ( input/ output  ratios,  percentages) 

Intra  group  communication 
Morale 

Number  of  patents  (issued,  applied  for) 

Number  of  presentations 

Number  of  publications  (books,  journals,  rates,  technical 
unpublished  manuscripts) 

Originality  (ideas,  studies,  program  formulation) 

Performance  appraisals 

Periodic  reviews 

Procurement  packages 

Progress  charts  (PERT,  GANTT) 

Quantification  of  qualitative  judgments 
Reputation 

Schedule  (milestones) 

Trends 

Work  unit  planning  document 


Consistent  theoretically  and  empirically  identified 
R&D  productivity  indicators  are  given  in  Table  3-7.  R&D 
productivity  evaluations  must  use  multiple  criteria  as 
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there  is  no  single  quantitative  measure  which  can  ade¬ 
quately  appraise  all  the  facets  of  a  research  and  develop¬ 
ment  effort.  Every  productivity  indicator  is  subject  to 
dispute.  The  criterion  for  inclusion  in  Table  3-7  is  that 
the  indicator  is  common  to  both  theoretical  and  empirical 
literature,  and  was  mentioned  at  least  four  times  in  the 
literature  reviewed. 

Contradictions .  Table  3-8  is  a  listing  of  the  productivity 
indicators  found  to  be  included  in  either  the  theoretical 
or  empirical  literature  and  also  refuted  in  either  the 
theoretical  or  empirical  literature.  For  example,  profit, 
profit  ratios,  and  market  share  may  be  good  measures  for 
some  industry  R&D  organizations,  but  do  not  apply  to  govern¬ 
ment  R&D  laboratories.  Preparation  of  work  statements  and 


TABLE  3-8 

CONTRADICTING  R&D  PRODUCTIVITY  INDICATORS  IDENTIFIED 
(OBJECTIVE  AND  SUBJECTIVE) 


Commercial  success 

Contract  monitoring  (weighting) 

Number  of  citations 

Number  of  publications  (weighting) 

Personnel  turnover 

Preparation  of  work  statements 

Profit 

Profit  ratios 
Quality 

Share  of  market 


contract  monitoring  are  contradicted  because  all  statements 
of  work  and  contracts  are  not  of  equal  value  m  time 
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requirements  and  in  dollar  value.  Various  weighting  schemes 
can  be  applied  to  these  types  of  indicators,  but  the 
weightings  are  not  common  to  all  laboratories  or  supported 
by  different  managers. 

Novelties .  There  are  nearly  as  many  novelty  R&D  productivity 
indicators  identified  in  Table  3-9  as  there  are  consistent 
indicators  in  Table  3-7.  The  novelty  indicators  are  cate¬ 
gorized  as  such  because  of  the  low  frequency  (3  or  fewer) 
of  occurrence  in  the  publications  reviewed.  The  majority 
of  indicators  were  mentioned  only  once  in  either  the  theore- 
.  tical  or  empirical  literature.  The  relatively  large  number 

of  indicators  are  indicative  of  the  concept  of  laboratory 
specificity  or  the  failure  of  other  laboratories  to  publish 
their  use  of  these  indicators. 

TABLE  3-9 

NOVELTY  R&D  PRODUCTIVITY  INDICATORS  IDENTIFIED 
(OBJECTIVE  AND  SUBJECTIVE) 


Amount  of  research  supervision  required 
Colleague  communication 
Hardware  software  specifications 
High  risk  versus  low  risk  projects 
Likeable  group  members 
Manufacturability 
Number  of  graduate  students 
Number  of  visitors 
Professional  society  evaluation 
Profit  contributions  resulting  from  goodwill 
created  by  the  research  establishment 
R&D  effort  ($)  to  maintain  existing  business  versus 
R&D  effort  in  quest  of  new  business 
Scientific  and  professional  society  memberships 
Sense  of  personal  achievement 
Technology  transitions 
University  evaluation 
Usefulness 

Variances  (schedule,  budget,  performance) _ 


CHAPTER  4 


DATA  COLLECTION 


Introduction 

The  third  area  of  interest  in  this  thesis  is  that 
of  identifying  the  state-of-the-art  of  productivity  measure¬ 
ment  criteria  (organizational  and  individual)  as  practiced 
by  both  DoD  and  selected  private  industry  laboratories. 

The  author  attempted  to  contact  all  DoD  laboratories  as 
identified  in  the  latest  available  DoD  in-House  RDT&E 
Activities  report  (69 :xxv-xxxi) .  This  reference  describes 
the  mission,  facilities,  and  funding  for  each  laboratory 
in  the  14  Air  Force,  34  Army,  and  22  Navy  laboratories  and 
centers.  Military  and  civilian  contacts  and  corresponding 
phone  numbers  were  obtained  from  the  September  1979  publica¬ 
tion  Institutional  Barriers  on  DoD  Laboratories:  A  Report 
of  the  Ad  Hoc  Task  Group  on  In-House  Laboratories  to  the 
Deputy  Under  Secretary  for  Research  and  Advanced  Technology 

(52:14-29).  An  extensive  telephone  survey  was  conducted. 

Responses  were  obtained  from  14  Air  Force  (100%) ,  30  Army 

(88%),  and  20  Navy  (91%)  laboratories  and  centers  for  an 

overall  DoD  response  rate  of  just  more  than  91%.  A  total 

of  21  industry  laboratories  were  interviewed  as  taken  from 

the  list  of  attendees  to  the  1980  International  Conference 

on  Productivity  Research  sponsored  by  the  Houston-based 

American  Productivity  Center  (APC) .  The  selections  were 
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made  on  the  basis  of  "most  likely  to  include  an  R&D 
laboratory,"  resulting  in  21  usable  responses.  The  21 
companies  clearly  are  not  representative  of  all  industry 
companies,  but  are  representative  of  those  44  "most  likely 
to  include  an  R4D  laboratory"  companies  identified  in  the 
aforementioned  listing. 

A  telephone  interview  methodology  was  chosen  for 
the  following  reasons: 

1.  A  higher  response  rate  can  be  anticipated. 

2.  A  telephone  interview  is  inherently  more 
personal  than  a  formal  questionnaire. 

3.  Additional  avenues  can  be  explored. 

4.  Ambiguities  are  minimized. 

5.  Responses  can  be  more  accurately  evaluated. 

6.  Investigator  familiarity  with -the  subject 
matter  (in  this  case  DoD  and  industry  laboratories) 
can  be  enhanced. 

7.  The  investigator  has  a  greater  degree  of  con¬ 
trol  in  choosing  the  level  of  laboratory  management 
to  interview. 

Each  interview  began  with  a  formal  introduction 
stating  the  investigator's  name,  school,  location,  state¬ 
ment  of  purpose,  how  the  information  would  be  used,  and  a 
request  for  voluntary  participation  in  the  thesis  effort. 

The  average  time  per  interview  was  approximately  25 
minutes.  Three  phone  calls  were  required  per  laboratory, 
on  the  average,  to  determine  the  proper  individual  to  talk 
to,  and  then  to  reach  that  individual.  The  telephone  inter¬ 
view  is  a  very  time  consuming  and,  thus,  demanding  data 
collection  methodology — singularly  the  greatest  disad¬ 
vantage. 

The  majority  of  interviews  revealed  there  is  a 

great  deal  of  interest  in  the  subject  of  R6D  laboratory 
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productivity,  a  fact  the  investigator  used  to  his  advantage 
while  questioning.  Each  interview  concluded  with  a  restate¬ 
ment  of  purpose,  how  the  information  would  be  used,  a  re¬ 
quest  to  be  included  on  the  thesis  distribution  list,  and 
a  request  for  permission  to  quote  (if  needed)  and  to  list 
the  company  (laboratory)  name  as  a  participant  in  the  thesis. 
Six  companies  preferred  anonymity  and  are  listed  as  such  in 
Table  A-4;  e.g.,  3  Company,  C  Company,  etc. 

Interview  Instrument 

The  telephone  interview  questionnaire  utilized 
(Appendix  B)  was  developed  to  obtain  a  maximum  amount  of 
information  for  the  eight  questions  solicited.  Productivity 
classification  (Q2a)  and  laboratory  productivity  indicators 
(Q3)  were  the  major  thrust  of  the  interview.  A  standardized 
response  form  was  developed  to  facilitate  note  taking  and 
to  reduce  the  interview  time  per  laboratory.  Response  con¬ 
fidentiality  was  emphasized  and  maintained  for  every  labora¬ 
tory. 

Question  1  intended  to  determine  which  laboratories 
had  any  published  productivity  studies  which  were  (are  being) 
conducted  in  the  respective  laboratories.  The  studies  may 
have  been  (be)  conducted  internally,  or  by  an  external  or¬ 
ganization  which  may  have  conducted  (conduct)  the  research 
through  a  thesis  effort,  or  by  contract,  or  by  higher  cor¬ 
porate  management. 
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The  productivity  classification  question,  Q2 ,  de¬ 
termined  whether  subjective  or  objective  indicators  were 


used  in  each  laboratory.  The  methods  used  to  determine  in¬ 
dividual  productivity  add  to  the  overall  knowledge  of  labora¬ 
tory  productivity  measurement. 

The  nature  of  specific  objective  and  subjective  in¬ 
dicators  were  identified  by  Q3.  This  question  is  the  primary 
thrust  of  the  interview.  After  the  initial  14  interviews 
were  conducted  with  the  Air  Force  laboratories, a  listing 
of  common  responses  emerged.  This  listing  became  a 
standardized  reference  asked  of  all  the  remaining  inter¬ 
viewees.  Air  Force  indicators  such  as  research  fellows, 
training/employee/year,  and  sick  leave  usage  are  not  in¬ 
cluded  because  these  did  not  emerge  until  late  in  the 
interviewing  process  and  were  therefore  not  asked  of  all 
Air  Force  laboratories.  The  18  objective  indicators  are  as 
follows: 


1.  Status  vs.  milestones; 

2.  Degree  technical  objectives  are  accom¬ 
plished.  (i.e..  Project  A  is  77%  complete); 

3.  Number  of  patents/year; 

4.  Number  of  publicatons/year; 

5.  Ratio  of  man  hours  expended/ re search 
dollar  expended; 

6.  Ratio  of  expected  sales/investment 
dollar  ratio; 

7.  Collective  determination  of  individual 
productivity; 

8.  Expenditures  vs.  budget; 

9.  Periodic  reviews; 

10.  Ratio  of  overhead/direct  costs; 

11.  Ratio  of  workload/researcher; 

12.  Research  fellows  contribution; 

13.  Laboratory  personnel  education  and 
credentials; 

14.  Training/employee; 

15.  Sick  leave  usage; 
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laboratory  productivity.  Efficiency  (output/input  ratios) 
and  effectiveness  (performance  of  mission)  daterminations 
were  specifically  asked  for  and  any  other  terminology  which 
may  apply  were  pursued. . 

Permission  to  list  the  laboratory  (company)  and 
permission  to  quote  (as  needed)  was  solicited  in  Q7.  The 
small  number  of  thesis  copies  available  for  distribution 
and  the  large  number  of  requests  for  copies  of  this  thesis 
prompted  the  investigator  to  volunteer  to  mail  the  Accession 
Document  (AD)  number  to  each  responding  laboratory  (company) 
(Q8).  An  accurate  formal  address,  name,  and  title  for  each 
respondent  was  compiled  for  this  purpose. 

Interview  Results 

The  majority  of  interview  results  are  presented  in 
Tables  4-1  to  4-8.  Both  the  actual  numbers  and  percentages 
are  given  in  each  block.  The  percentage  values  are  rounded 
to  the  nearest  1%,  accounting  for  any  summation  variances 
from  100%.  The  interview  responses  do  not  warrant  further 
statistical  analysis  as  is  common  with  questionnaires  soli¬ 
citing  responses  on  an  ordinal  scale;  e.g.  1-7.  The  Tables 
of  data  and  discussions  are  presented  in  the  same  order  the 
interview  questions  were  asked. 

Table  4-1  reflects  the  response  rates  for  each  cate¬ 
gory  of  laboratories  resulting  in  an  overall  response  ratio 
of  75%,  which  is  indeed  higher  than  many  written  question¬ 
naire  response  rates  (38:32;  65:43;  12:23;  16:63;  56:32). 

The  private  industry  responae  rate  was  determined  on  the 

89 


LABORATORY  RESPONSE  RATIO  AND  LEVEL  OF  LABORATORY  MANAGEMENT 


basis  of  21  responses  out  of  44  companies  "most  likely  to 
include  an  R&D  laboratory",  as  enumerated  in  a  listing  of 
participants  to  the  International  APC  Conference.  Removing 
the  industry  figures  result  in  just  over  a  91%  response  rate 
for  all  DoD  laboratories,  a  statistic  reflective  of  labora¬ 
tory  interest.  Additional  private  companies  were  not  pur¬ 
sued  due  to  the  time  constraints  of  this  research.  An 
attempt  to  keep  all  responses  from  the  same  level  of  labora¬ 
tory  management  was  not  possible  due  to  vacations,  travel 
duty,  and  referrals.  Therefore,  a  record  of  the  respon¬ 
dent's  laboratory  management  level  was  made  and  resulted  in 
fairly  high  upper  management  response  (85%) ,  a  low  middle 
management  response  rate  (14%)  ,  and  only  one  lower  manage¬ 
ment  response  (1%) .  This  information  is  important  be¬ 
cause  the  perceptions  of  laboratory  productivity  may  not 
be  the  same  at  the  different  management  levels  in  a  labora¬ 
tory. 

Table  4-2  shows  the  response  data  relative  to  produc¬ 
tivity  measurement  studies  currently  underway  in  addition 
to  the  studies  which  have  been  conducted  either  internally 
or  externally  since  1970.  All  government  laboratories  are 
collectively  considered.  The  data  are  not  completely  ac¬ 
curate  due  to  the  wording  of  the  question,  i.e.,  "Are  you 
aware  of  .  .  .  The  respondent  may  not  have  been  aware  of 
all  productivity  measurement  studies  since  1970,  and  there 
are  at  least  four  cases  where  the  respondent  voluntarily 

admitted  to  less  than  one  year  occupancy  in  his  present 
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management  position.  Responses  relative  to  current  ongoing 
research  is  considered  accurate. 

Table  4-3  clearly  reveals  that  the  greater  majority 
of  all  R&D  laboratories  incorporate  a  combination  of  subjec¬ 
tive  and  objective  criteria  in  determining  their  productivity. 
Only  four  industry  laboratories  did  not  determine  laboratory 
productivity  due  to  the  complexity  of  the  task.  The  results 
further  emphasize  the  point  that  objective  indicators  are 
not  easily  developed  as  indicated  by  the  higher  percen¬ 
tage  of  laboratories  relying  on  subjective  determinations 
when  both  subjective  and  objective  criteria  are  not  used. 

Individual  researcher  productivity  was  unanimously 
determined  at  least  in  part  on  the  basis  of  performance  ap¬ 
praisals.  The  various  performance  appraisal  systems  are  as 
follows : 

1.  General  Performance  Appraisal  -  Industry;. 

2.  OER  -  Officer  Efficiency  Report  -  DoD; 

3.  GMAS  -  General  Managers  Appraisal  System 
(1  JUN  81)  -  AF; 

4.  JPAS  -  Job  Performance  Appraisal  System 
(1  OCT  81)  -  AF; 

5.  MPS  -  Merit  Pay  System  -  Army  and  Navy; 

6.  GPAS  -  General  Performance  Appraisal 
System  -  Army  - 

These  systems  incorporate  the  concepts  of  Management 
by  Objectives  (MBO)  (1:83)  and  a  participative  leadership 
style  (1:391-392).  Additional  individual  productivity 
criteria  include,  but  are  not  limited  to  the  following: 

1.  The  number  of  novel  ideas  generated; 

2.  The  number  of  patents; 

3.  Patent  submissions,  and  invention 
disclosures ; 
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4.  The  workload  and  quality  of  projects; 

5.  The  number  of  original  publications 
produced  both  in-house  and  contributed  to  the 
public  literature; 

6.  The  number  of  publications  in  referee 
journals ; 

7.  Peer  quality  reviews; 

8.  The  achievement  of  target  and  overtarget 
objectives ; 

9.  The  number  and  quality  of  presentations 
given ; 

10.  The  number  of  conferences  attended; 

11.  Professional  society  involvement  and 
recognition; 

12.  The  level  of  effort  expended; 

13.  project  salesmanship  and  visibility  for 
funding  support; 

14.  periodic  (every  three- to-six  months) 
reviews ; 

15.  performance  within  the  budget; 

16.  Variances  (cost  and  schedule) ; 

17.  critical  path  performance; 

18.  Value  added  per  labor  hour; 

19.  problems  solved. 

Table  4-4  shows  the  results  of  Q3a  (Appendix  B) , 
the  objective  indicators  of  R&D  productivity.  Objective 
indicators  most  common  to  industry  and  government  R&D  are 
status  vs.  milestones,  degree  technical  objectives  are 
reached,  expenditures  vs.  budget  and  periodic  reviews. 
Predominantly,  objective  indicators  of  R&D  productivity 
in  the  government  include  the  number  of  publications/year , 
collective  determination  of  individual  productivity,  re¬ 
search  fellows  contributions,  laboratory  personnel  educa¬ 
tion  and  credentials,  training/employee,  sick  leave  usage, 
technical  transitions  (both  military  to  civilian  applica¬ 
tion  and,  e.g.  basic  research  to  advanced  research),  pro¬ 
blems  identified  and  solved,  and  consultations  to  organiza¬ 
tions  external  to  the  laboratory.  The  only  objective 
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TABLE  4-4  (Cont'd) 

INTERVIEW  RESPONSES  RE:  OBJECTIVE  INDICATORS  OF  R&D  PRODUCTIVITY  (ORGANIZATIONAL) 


indicator  of  R&D  productivity  in  industry  alone  is  expected 
sales/investment  dollar  ratio,  which  does  not  apply  to 
government  R&D.  Less  important  objective  indicators  of 
R&D  productivity  common  to  both  industry  and  government 
R&D  are  the  number  of  patents/year,  the  ratio  of  man-hours 
expended/research  dollar  expended,  the  overhead/direct  cost 
ratio,  and  the  workload/researcher  ratio. 

Nearly  every  respondent  commented  that  at  least 
one  of  the  18  standardized  indicators  was  measured  but  not 
for  the  purpose  of  measuring  laboratory  productivity.  For 
example,  the  ratio  of  overhead/direct  costs  may  indeed  be 
tracked,  but  it  is  not  used  as  an  indicator  of  laboratory 
productivity  because  the  overhead  costs  are  not  within  the 
control  of  the  laboratory.  Therefore,  it  becomes  necessary 
to  emphasize  to  each  respondent  to  indicate  whether  or 
not  each  of  the  18  standardized  and  any  additional  indi¬ 
cators  are  used  in  determining  laboratory  productivity. 

As  a  result,  all  objective  indicator  responses  are  uniform 
in  that  they  reflect  actual  measurements  taken  to  determine 
laboratory  productivity  but  may  also  be  used  in  other  report¬ 
ing  documents. 

Additional  objective  indicators  of  R&D  productivity 
not  included  in  the  table  are  as  follows: 

1.  Project  output/ investment  cost; 

2.  Personnel  commitment  rates; 

3.  Obligation  rates; 

4.  Disbursement  rates; 

5.  Fieldings  (equipment  or  techniques 
which  are  incorporated  into  operational  use) ; 

6.  The  in/house/out-house  ratio; 


7.  Man-hours  expended; 

8.  The  number  of  presentations  given; 

9.  Workload/research  department; 

10.  New  vaccines; 

11.  Treatments; 

12.  Diagnostic  technical  advancements. 

These  objective  productivity  indicators  were  not  common 
enough  to  tabulate,  but  are  worthy  of  mention.  The  last 
three  indicators  are  clearly  usable  for  a  medical  R&D 
laboratory  but  do  not  apply  to  other  R&D  laboratories. 

This  represents  a  good  example  of  the  inability  to  develop 
universal  laboratory  productivity  indicators  without  making 
these  indicators  overly  general.  The  unique  inherent  mis¬ 
sions,  personnel,  and  facilities  of  each  laboratory  tend 
to  indicate  that  a  specific  set  of  indicators  should  be 
developed  to  account  for  the  uniqueness  of  each  type  of 
laboratory  (medical,  electronic,  armament,  etc.),  if  not 
for  each  laboratory  itself.  In  addition  to  these  objective 
indicators,  several  subjective  indicators  were  identified. 
These  subjective  indicators  are  teamwork  (the  ability  of 
a  group  to  work  together) ,  laboratory  reputation,  and 
sponsor  feedback  (feedback  from  the  organization  sponsor¬ 
ing  the  research) . 

A  subjective  determination  of  laboratory  produc¬ 
tivity  was  found  to  occur  in  all  but  four  (17  of  21)  indus¬ 
try  laboratories.  Table  4-5  reveals  that  R&D  laboratories 
perceive  their  productivity  to  be  determined  more  by  upper 
laboratory  management  and  higher  corporate  management  than 
by  each  laboratory  management  level.  Industry  responses 
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TABLE 


indicate  that  a  subjective  evaluation  of  laboratory  pro¬ 
ductivity  is  roughly  equivalent  at  all  three  levels,  and 
differs  dramatically  with  the  overall  government  figures. 
Navy  responses  place  greater  perceived  subjective  labora¬ 
tory  determinations  at  the  higher  corporate  management 
level  while  both  Air  Force  and  Army  subjective  laboratory 
determinations  are  perceived  to  occur  more  frequently  at 
the  conunander/director /division  chief  level.  The  Air 
Force  response  rate  of  29%  at  the  higher  corporate  manage¬ 
ment  level  is  extremely  low  when  compared  to  Army  (77%) 
and  Navy  85%)  responses,  but  part  of  this  large  variation 
may  be  attributable  to  the  fact  that  the  Air  Force  labora¬ 
tories  were  the  first  set  of  laboratories  interviewed  and, 
as  such,  may  reflect  some  effect  from  the  formulation  of 
the  standardized  list.  The  percentages  do  not  sum  to  100% 
because  many  of  the  laboratories  gave  multiple  responses, 
indicating  the  level  of  determination,  and  that  determina¬ 
tions  are  also  made  at  other  levels. 

Table  4-6  shows  the  sources  of  new  R&D  projects; 
the  specific  approaches  to  be  taken  to  comply  with  higher 
corporate  guidance  (directives) .  No  single  management 
level  is  dominant.  The  high  industry  (86%)  and  government 
(100%)  values  for  the  technical  and  professional  recommen¬ 
dations  category  represent  a  strong  reliance  on  and  an 
effective  use  of  laboratory  personnel  in  project  selection. 
Government  figures  for  special  committees  (86%) ,  user  iden¬ 
tification  (75%)  ,  and  consultants  (56%)  average  over  21% 
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higher  than  the  respective  industry  figures  (67%,  57%,  29%). 
The  industry  laboratory  value  for  upper  management  (38%) 

doubles  the  government  percentage  (19%)  indicating  that 
the  government  methodology  for  selecting  new  R&D  projects 

is  less  centralized  than  the  method  used  in  industry. 

Special  committees  cure  the  second  most  reported 
sources  of  new  R&D  projects  (industry  -  67%,  government  - 
86%) .  Commonly  used  project  selection  committee  names  are 
as  follows: 

1.  Corporate  Board; 

2.  Corporate  Planning  Group; 

3.  Executive  Program  Council; 

4.  Internal  Quality  Control  Group; 

5.  Proposal  Review  Board; 

6.  Research  Advisory  Council  (RAC); 

7.  Research  Planning  Committee  (RPC); 

8.  Steering  Committee; 

9.  Strategic  Planning  Group; 

10.  Technical  Advisory  Committee  (TAC) ; 

11.  Technological  Base  Review  Committee, 

12.  Technology  Center  Management  Committee; 

13.  Technical  Directorate  Group; 

14.  Technical  Forecasting  Program  Committee; 

15.  technical  Planning  Board  TPB; 

16.  Technical  Thrust  Committee. 

The  committees  typically  evaluate  on-going  programs,  proposed 
new  projects,  contract  proposals,  and  overall  mission  objec¬ 
tives.  User  identification  of  programs  (marketing  in  industry 
and  operational  commands  in  DoD)  is  the  third  most  reported 
source  of  new  R&D  projects  and  consultants  (external  to  the 
laboratory)  are  the  fourth  most  reported  source  of  R&D  pro¬ 
jects.  The  percentages  indicate  that  there  is  an  effective 
use  of  laboratory  personnel  and  external  references. 

The  utility  of  laboratory  productivity  measurements 
are  revealed  in  Table  4-7.  A  self  comparison  as  a  function 
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UTILITY  OF  PRODUCTIVITY  MEASUREMENTS 


of  time  is  the  main  purpose  of  making  productivity  measure¬ 
ments.  Only  15  respondents  indicated  there  was  no  compari¬ 
sons  made  at  all,  and  six  of  these  laboratories  do  not  use 
objective  indicators.  Competitive  comparisons  are  an 
important  use  of  productivity  measurements,  but  many 
respondents  pointed  out  that  it  is  a  very  difficult  task 
to  obtain  competitive  data  with  which  to  make  such  com¬ 
parisons.  The  utility  of  productivity  measurements 
to  improve  individual  motivation,  examine  the  ratio  of 
technical  objectives  accomplished/annual  cost,  and  the 
allocation  of  resources  and  mission  alignment  are  also 
important  uses. 

Self  comparisons  as  a  function  of  time  are  con¬ 
sistently  higher  for  government  laboratories  than  for  in¬ 
dustry  laboratories.  Competitive  comparisons  are  roughly 
equivalent  (in  percentage)  for  all  laboratories,  with  Air 
Force  laboratories  lagging  slightly  behind  the  average 
(29%  vs.  42%).  The  government  utility  of  productivity 
measurements  to  improve  individual  motivation  nearly 
triples  the  industrial  utility  (69%  vs.  29%).  Of  the 
government  laboratories ,  Air  Force  laboratories  do  not 
use  productivity  measurements  to  improve  individual  moti¬ 
vation  to  the  extent  that  Army  and  Naval  laboratories 
use  them  (7%  vs.  80%  and  95%).  This  difference  may  be 
partly  explained  by  the  higher  percentage  of  Air  Force 
laboratories  found  not  to  make  any  productivity  comparisons 

at  all  (Air  Force  -  29%,  Array  -  7%,  Navy  -  15%).  The  29% 
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of  industry  laboratories  making  no  productivity  comparisons 
just  more  than  doubles  the  14%  of  government  laboratories 
not  making  productivity  comparisons .  The  utility  of  pro¬ 
ductivity  measurements  to  determine  the  ratio  of  technical 
objectives  accomplished/ annual  cost  ranges  from  a  low  of 
48%  to  a  high  of  60%  for  all  laboratories,  indicating  a 
similar  distribution  for  this  category.  The  utility  of 
productivity  measurements  to  determine  the  allocation  of 
resources  Is  150%  higher  in  government  laboratories 
than  in  industry  laboratories  (20%),  with  Navy  laboratories 
leading  the  rest  (60%) . 

Table  4-8  data  reflects  that  the  effectiveness  com¬ 
ponent  of  productivity  is  determined  more  often  than  ef¬ 
ficiency  in  government  R&D  and  in  the  private  companies 
interviewed.  Because  of  the  nature  of  R&D,  the  laboratory 
responses  indicate  a  higher  priority  to  determine  mission 
accomplishment  over  efficiency.  Army  and  Navy  laboratory' 
percentages  for  efficiency  and  effectiveness  are  signifi¬ 
cantly  higher  than  those  values  for  Air  Force  laboratories , 
and  correspondingly,  the  percentage  of  Air  Force  labora¬ 
tories  using  neither  efficiency  nor  effectiveness  as  compo¬ 
nents  of  productivity  is  significantly  different  than  the 
Army  and  Navy  laboratory  values.  Other  productivity  compo¬ 
nents  which  are  used  either  instead  of  or  in  addition  to 
efficiency  and  effectiveness  are  also  listed. 
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Selected  Quotations 

Several  comments  concerning  various  aspects  of  R&D 
productivity  were  made  during  the  telephone  interviews 
which  could  not  be  properly  tabularized.  These  comments 
are  included  because  they  often  reflect  multiple  respondent 
feelings  and  offer  insight  to  the  problem  of  R&D  produc¬ 
tivity  measurement. 


"Technical  reports,  journal  articles,  in-house 
publications  and  presentations  to  scientific  con¬ 
ferences  are  used  as  a  good  measure  of  organiza¬ 
tional  productivity  and  quality  of  product." 

"Subjectively,  a  partial  indication  of  labora¬ 
tory  productivity  is  given  by  the  successful  im¬ 
plementation  into  operational  use." 

"There  is  no  universal  formula  to  precisely 
measure  R&D  productivity.  I  feel  it  is  better  to 
evaluate  productivity  based  on  both  quantitative 
and  qualitative  factors  rather  than  try  to  strictly 
measure  productivity  with  precise  quantifications. 

A  situation-specific  evaluation  method  should  be 
established  by  each  sector  manager.  The  purposes  of 
this  qualitative  and  quantitative  criteria  are 
1)  to  evaluate  sector  productivity,  and  2)  to  re¬ 
port  the  results  to  the  next  management  level.  Once 
the  historical  base  is  established,  the  emphasis 
should  be  to  monitor  and  improve  productivity." 

"R&D  laboratory  productivity  indicators  aid 
the  company  in  determining  whether  or  not  the  right 
people  are  being  hired." 

"...  defines  productivity  as  the  ratio  of 
Output  of  Valuable  Projects  to  Investment  Cost. 
Productivity  Indicators  are  the  ratio  of  R&D  dol¬ 
lars  to  Revenue  dollars,  and  the  project's  value 
to  the  user  organization." 

"Productivity  indicators  permit  the  identi¬ 
fication  of  both  organizational  weaknesses  and 
strengths,  and  aid  in  determining  how  best  to 
allocate  dollar,  personnel,  and  faciliate  re¬ 
sources.  " 
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"Laboratory  leadership  is  most  important 
for  motivating  people  and  keeping  on  top  of  tech¬ 
nology.  Salesmanship  and  visibility  are  addi¬ 
tional  factors  considered  in  establishing  a 
project's  funding  level." 

"Progress  in  development  can  be  more  easily 
measured  quantitatively  than  basic  research  which 
is  necessarily  more  subjective." 

"Productivity  measurements  help  laboratory 
management  determine  the  projects  which  will  yield 
the  best  product  possible  for  the  user,  to  de¬ 
termine  projects  with  the  best  ROI." 

"Productivity  measurements  are  used  in  part  to 
determine  the  efficiencies  of  our  functional 
managers  in  accomplishing  the  annually  increasing 
number  of  tasks." 

".  .  .  it  is  not  appropriate  to  apply  pro¬ 
ductivity  to  direct  labor  in  R&D  laboratories , 
and  that  the  emphasis  should  be  on  what  we  are 
doing  and  are  we  making  any  progress." 

"R&D  productivity  indicators  should  be  used 
to  best  determine  resource  allocation,  organiza¬ 
tional  alignment,  and  for  supervisory  performance 
evaluations . " 

"R&D  laboratory  productivity  measures  help 
determine  which  new  projects  should  be  investi¬ 
gated,  which  projects  should  be  terminated,  the 
allocation  of  resources,  and  identify  the  pro¬ 
ductive  and  nonproductive  individuals  in  the 
organization. " 

"...  the  operational  commands  need  a  more 
direct  mechanism  to  communicate  their  needs  to 
the  laboratories  for  small  projects  and  problems." 

"One  major  component  of  productivity  in  our 
laboratory  is  the  quality  of  the  scientific 
journals  which  accept  our  articles,  indicating 
the  quality  of  our  publications." 

"Many  objective  indicators  have  a  high  degree 
of  inherent  subjectivity  and  these  indicators  de¬ 
pend  to  a  large  degree  on  the  management  style 
and  management  goals  established  in  each  indi¬ 
vidual  laboratory." 
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Summary 

A  telephone  interview  methodology  was  chosen  to 
determine  the  state-of-the-art  of  productivity  management 
in  21  industry,  14  Air  Force,  30  Army,  and  20  Navy  labora¬ 
tories,  resulting  in  an  overall  response  rate  of  75%  and 
91%  for  DoD  laboratories.  The  interview  results  are  tabu¬ 
lated  and  interpreted  for  each  question.  Upper  laboratory 
management  provided  85%  of  all  responses. 

The  majority  of  laboratories  (government  -  92%,  in¬ 
dustry  -  62%)  use  both  subjective  and  objective  measures  of 
productivity.  Individual  productivity  measurements  are  based 
on  a  form  of  performance  appraisal.  Status  vs.  milestones, 
the  degree  technical  objectives  are  reached,  expenditures 
vs.  budget,  and  periodic  reviews  were  the  most  common  (of 
18)  indicators  to  both  government  and  industry  laboratories. 
Additional  indicators  were  identified.  The  use  of  technical 
and  professional  recommendations,  special  committees,  cus¬ 
tomers,  and  external  consultants  indicates  the  effective 
use  of  laboratory  response.  Self  comparisons  and  competi¬ 
tive  comparisons  are  the  primary  purposes  of  productivity 
measurements.  Effectiveness  is  a  more  important  component 
of  productivity  than  efficiency  is.  Interviewee  comments 
offer  insight  to' the  problems  of  R&D  productivity  measure¬ 
ment  and  provide  additional  information  which  cannot  be 


easily  tabulated. 


CHAPTER  5 


Summary  of  Research 

This  investigation  into  the  state-of-the-art  of 
productivity  measurement  in  the  research  and  development 
community  consisted  of  exploration  into  three  distinct 
areas  of  interest: 

1.  A  review  of  the  theoretical  literature  of  in¬ 
dustry  and  DoD  laboratories. 

2.  A  review  of  the  empirical  literature  of  in¬ 
dustry  and  DoD  laboratories. 

3.  An  extensive  telephone  interview  of  industry 
and  DoD  laboratories. 

These  three  interests  were  pursued  using  two  methodologies: 
an  exhaustive  search  of  the  theoretical  and  empirical  litera¬ 
ture,  and  an  extensive  telephone  interview  process  conducted 
with  64  DoD  and  21  industry  laboratories.  R&D  productivity 
and  its  components  were  defined  and  consistently  used  through¬ 
out  this  thesis.  The  literature  review  identified  the  objec¬ 
tive  and  subjective  R&D  productivity  indicators  both  proposed 
and  utilized  from  1960  to  the  present. 

The  study  limited  its  focus  to  the  measurement  of 
R&D  productivity.  Productivity  improvement  and  factors  af¬ 
fecting  laboratory  productivity  were  not  explored.  To  serve 
as  an  initial  step  for  follow-on  study,  references  for  these 
related  topics  are  included  in  the  expanded  Related  Sources 
of  the  Bibliography.  Theoretical  and  empirical  literature  of 
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R&D  productivity  indicator  similarities,  contradictions, 
and  novelties  were  identified. 


Summary  of  Results 

A  chronological  listing  of  13  theoretical  and  41 
empirical  publications  were  listed  in  Appendix  C  and  pre¬ 
sented  in  Chapter  3.  Twenty-one  consistently  identified 
similar  R&D  productivity  indicators,  ten  contradictory  in¬ 
dicators,  and  17  novelty  indicators  were  identified  from 
the  literature  in  Tables  3-7,  3-8,  and  3-9.  Consistent 
indicators  were  those  indicators  found  to  be  common  to  both 
the  theoretical  and  empirical  literature  and  were  mentioned 
at  least  four  times.  Contradictory  indicators  were  those 
indicators  found  to  be  included  in  either  the  theoretical 
or  empirical  literature  and  also  refuted  in  either  the 
theoretical  or  empirical  literature.  Novelty  indicators 
were  the  indicators  found  to  occur  three  or  fewer  times  in 
the  publications  reviewed. 

The  telephone  interview  instrument  was  used  to  sur¬ 
vey  21  industry,  14  Air  Force,  30  Army,  and  20  Navy  labora¬ 
tories  resulting  in  an  overall  response  rate  of  75%  and  91% 
for  DoD  laboratories.  Upper  laboratory  management  provided 
85%  of  all  responses.  Individual  productivity  measurements 
are  based  on  a  form  of  performance  appraisal.  Status  versus 
milestones,  the  degree  technical  objectives  are  reached, 
expenditures  versus  budget,  and  periodic  reviews  were  the 
most  common  (of  18)  indicators  to  both  government  and 


industry  laboratories.  The  use  of  technical  and  pro¬ 
fessional  recommendations,  special  committees,  customers, 
and  external  consultants  indicates  the  effective  use  of 
laboratory  resources.  Self  comparisons  and  competitive 
comparisons  are  the  primary  purpose  of  productivity  measure¬ 
ments.  Effectiveness  is  perceived  as  a  more  important  com¬ 
ponent  of  productivity  than  efficiency  is.  Interviewee 
comments  offer  further  insight  into  the  problems  of  R&D 
productivity  measurement. 

The  majority  of  laboratories  interviewed  (govern¬ 
ment  -  92%,  industry  -  62%)  used  both  subjective  and  ob¬ 
jective  measures  of  productivity.  Combined  subjective  and 
objective  indicators  were  suggested  by  46%  of  the  theoreti¬ 
cal  "combined  government  and  industry"  category  and  by  59% 
of  the  empirical  "combined  government  and  industry"  cate¬ 
gory.  The  incidence  of  laboratories  utilizing  both  subjec¬ 
tive  and  objective  measures  was  clearly  dominant  for  the 
literature  and  for  the  surveyed  laboratories,  but  less  domi¬ 
nant  for  the  literature.  When  "subjective  only"  and  "objec¬ 
tive  only"  categories  are  examined,  "objective  only"  indi¬ 
cators  were  preferred  in  the  literature,  whereas  the  sub¬ 
jective  only"  indicators  were  the  preference  of  the  inter¬ 
viewed  laboratories.  Of  the  18  standardized  interview  in¬ 
dicators,  training  per  employee,  problems  solved/identified, 
consultations  to  outside  organizations,  and  workload  per  re¬ 
searcher  were  not  identified  in  the  literature.  The 

"research  fellows  contributions"  interview  indicator  was 
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found  to  be  a  novelty  indicator  in  the  literature.  All 
remaining  interview  indicators  were  classified  as  con¬ 
sistent  literature  indicators. 


Recommendations 


The  following  recommendations  evolved  out  of  the 
results  of  this  research  and  other  research  on  productivity 
measurement  in  R&D  laboratories. 

1.  Duplicate  the  interview  to  validate  the  find¬ 
ings  of  the  present  study.  The  literature  review 
should  also  be  updated  to  reflect  the  most  current 
theories  and  the  most  recent  results  of  empirical 
inves  tigations . 

2.  An  expanded  literature  search  may  be  enhanced 
by  taking  full  advantage  of  the  National  Technical 
Information  Service  listing,  for  which  a  fee  is  charged. 

3.  Consider  the  investigation  of  R&D  productivity 
improvement  rather  than  the  direct  measurement  of  R&D 
productivity.  Methods  for  productivity  improvement 
and  factors  affecting  R&D  productivity  are  more  readily 
available  and  may  be  more  beneficial. 


115 


APPENDICES 


116 


APPENDIX  A 

DEPARTMENT  OF  DEFENSE  RESEARCH  AND  DEVELOPMENT  LABORATORIES 
AND  PARTICIPATING  INDUSTRY  LABORATORIES 
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TABLE  A-l 


AIR  FORCE  RESEARCH  AND  DEVELOPMENT 
LABORATORIES 


AEROPROPULSION  LABORATORY 

AEROSPACE  MEDICAL  RESEARCH  LABORATORY 

ARMAMENT  DEVELOPMENT  AND  TEST  CENTER 

AVIONICS  LABORATORY 

ENGINEERING  SERVICES  CENTER/RD 

FLIGHT  DYNAMICS  LABORATORY 

FRANK  J.  SEILER  RESEARCH  LABORATORY 

GEOPHYSICS  LABORATORY 

HUMAN  RESOURCES  LABORATORY 

MATERIALS  LABORATORY 

ROCKET  PROPULSION  LABORATORY 

ROME  AIR  DEVELOPMENT  CENTER 

SCHOOL  OF  AEROSPACE  MEDICINE 

WEAPONS  LABORATORY 


TABLE  A- 2 

NAVY  RESEARCH  AND  DEVELOPMENT  LABORATORIES 


AEROSPACE  MEDICAL  RESEARCH  LABORATORY 

AIR  DEVELOPMENT  CENTER 

BIOSCIENCES  LABORATORY 

CIVIL  ENGINEERING  LABORATORY 

CLOTHING  AND  TEXTILE  RESEARCH  FACILITY 

COASTAL  SYSTEMS  CENTER 

DAVID  W.  TAYLOR  NAVAL  SHIP  R&D  CENTER 

DENTAL  RESEARCH  INSTITUTE,  NTC 

ENVIRONMENTAL  PREDICTION  RESEARCH  FACILITY 

HEALTH  RESEARCH  CENTER 

MEDICAL  RESEARCH  INSTITUTE 

MEDICAL  RESEARCH  UNIT  NO.  2 

MEDICAL  RESEARCH  UNIT  NO.  3 

OCEAN  R&D  ACTIVITY 

OCEAN  SYSTEMS  CENTER 

PERSONNEL  RESEARCH  &  DEVELOPMENT  CENTER 
RESEARCH  LABORATORY 

SUBMARINE  MEDICAL  RESEARCH  LABORATORY 
SURFACE  WEAPONS  CENTER 
UNDERSEAS  RANGES  DEPARTMENT 
UNDERWATER  SYSTEMS  CENTER 
WEAPONS  CENTER 


TABLE  A- 3 


ARMY  RESEARCH  AND  DEVELOPMENT  LABORATORIES 


AEROMEDICAL  RESEARCH  LABORATORY 
ATMOSPHERIC  SCIENCES  LABORATORY 
AVIATION  R&T  LABORATORY 
AVIONICS  R&D  ACTIVITY 
BALLISTIC  RESEARCH  LABORATORY 
CHEMICAL  SYSTEMS  LABORATORY 
COLD  REGIONS  R&E  LABORATORY 

COMBAT  SURVEILLANCE  &  TARGET  ACQUISITION  LABS 

COMMUNICATIONS  ELECTRONICS  COMMAND 

CONSTRUCTION  ENGINEERING  RESEARCH  LABORATORY 

ELECTRONIC  WARFARE  LABORATORY 

ELECTRONICS  TECHNOLOGY  &  DEVICES  LABORATORY 

ENGINEER  TOPOGRAPHIC  LABORATORIES 

ENGINEERING  WATERWAYS  EXPERIMENT  STATION 

FIRE  CONTROL  &  SMALL  CALIBER  WEAPONS  SYSTEMS  LAB 

HARRY  DIAMOND  LABORATORIES 

HUMAN  ENGINEERING  LABORATORY 

INSTITUTE  OF  DENTAL  RESEARCH 

INSTITUTE  OF  SURGICAL  RESEARCH 

LARGE  CALIBER  WEAPONS  SYSTEMS  LABORATORY 

LETTERMAN  ARMY  INSTITUTE  OF  RESEARCH 

MATERIALS  AND  MECHANICS  RESEARCH  CENTER 

MEDICAL  BIOENGINEERING  R&D  LABORATORY 

MEDICAL  RESEARCH  INSTITUTE  OF  CHEMICAL  DEFENSE 

MEDICAL  R&D  COMMAND 

MEDICAL  RESEARCH  INSTITUTE  OF  INFECTIOUS  DISEASES 

MISSILE  R&D  COMMAND 

MOBILITY  EQUIPMENT  R&D  COMMAND 

NATICK  R&D  COMMAND 

NIGHT  VISION  &  ELECTRO-OPTICS  LABORATORY 
RESEARCH  INSTITUTE  OF  ENVIRONMENTAL  MEDICINE 
SIGNALS  WARFARE  LABORATORY 
TANK-AUTOMOTIVE  R&D  COMMAND 
WALTER  REED  ARMY  INSTITUTE  OF  RESEARCH 


TABLE  A- 4 


PARTICIPATING  INDUSTRY  LABORATORIES 


ALUMINUM  COMPANY  OF  AMERICA 
AMERICAN  CAN  COMPANY 
AMERICAN  TELEPHONE  AND  TELEGRAPH 
ATLANTIC  RICHFIELD  COMPANY 
B  COMPANY 
C  COMPANY 

CELANESE  CHEMICAL  RESEARCH  COMPANY 
CITIES  SERVICE  COMPANY 
CORNING  GLASS  WORKS 
D  COMPANY 

EXXON  CHEMICALS  AMERICA 
F  COMPANY 

GENERAL  MILLS,  JAMES  FORD  BELL  TECHNICAL  CENTER 
HUGHES  AIRCRAFT  COMPANY 
I  COMPANY 
J  COMPANY 

KAISER  ALUMINUM  AND  CHEMICAL  COMPANY 
PHILLIPS  PETROLEUM  COMPANY 
STRATEGIC  PLANNING  INSTITUTE 
UNION  PACIFIC  RAILROAD 
j  UNITED  STATES  STEEL 
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APPENDIX  B 

DATA  COLLECTION  INSTRUMENT 


TELEPHONE  INTERVIEW  QUESTIONNAIRE 


ORGANIZATION 

NAME 

POSITION  TITLE 

LABORATORY  MANAGEMENT  LEVEL 


Ql.  Are  you  aware  of  any  internal  or  external  productivity 
measurement  studies  that  have  been  conducted  in  your  labora 
tory  since  1970?  Are  there  currently  any  internal  or  ex¬ 
ternal  studies  in  progress? 

Q2 .  PRODUCTIVITY  CLASSIFICATION 

a.  Is  your  laboratory's  productivity  determined  sub¬ 
jectively,  by  objective  indicators,  both  subjectively 
and  objectively,  or  not  at  all? 

b.  How  does  your  laboratory  determine  individual  re¬ 
searcher  productivity? 

Q3 .  LABORATORY  PRODUCTIVITY  IND ICATOkS/DETERMINATI ON 

a.  What  objective  productivity  indicators  (criteria, 
standard,  formula)  are  used  in  your  laboratory? 

b.  Is  your  laboratory's  productivity  subjectively 
determined  by  the  laboratory  director  (commander) , 
division  managers,  or  at  each  management  level? 
What  are  the  subjective  criteria? 

Q4.  This  question  concerns  the  sources  of  new  R&D  projects 
How  are  the  specific  new  projects  (approaches)  selected? 

Is  there  a  primary  management  level  for  project  selection? 
Do  you  rely  on  the  recommendation  of  your  technical  and 
professional  personnel,  and  user  and  industry?  Do  you 
normally  use  special  committees  or  consultants  for  pro¬ 
ject  selection? 

Q5.  This  question  concerns  the  utility  of  productivity  in¬ 
formation.  Is  your  laboratory's  productivity  compared 
against  itself  as  a  function  of  time  or  against  other  lab¬ 
oratories?  Is  laboratory  productivity  used  to  improve  in¬ 
dividual  motivation?  Does  your  laboratory  use  some  form 
of  the  ratio  of  accomplishments/annual  cost?  Other  pur¬ 
poses? 
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Q6.  Is  laboratory  effectiveness  (performing  laboratory 
mission)  and  efficiency  (output/input  ratios)  specifically 
determined?  Are  there  other  terms  used  as  components 
of  productivity? 

Q7.  As  I  stated  in  my  introduction,  this  information  is 
to  be  used  in  tabled  form  such  that  your  responses  cannot 
be  correlated  with  you  or  your  laboratory.  Do  I  have  your 
permission  to  list  your  company  (laboratory)  as  a  parti- 
pant  company  (laboratory)?  May  I  quote  your  responses? 

(if  needed)? 

Q8.  Would  you  like  to  be  included  on  the  distribution 
list  for  this  thesis?  Formal  Address? 
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APPENDIX  C 

CHRONOLOGICALLY  TABLED  LITERATURE 
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THEORETICAL  LITERATURE  REVIEW 
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KEY«  C  -  Cases;  Co  ■  Complete!  F  »  Field)  G  «  Government!  I  »  Industry;  In  »  Interview; 

LR  •  Literature  Review;  M  «  Model;  N  “  None;  0  -  Objective;  P  »  Percentage;  S  -  SubjecLive 
SD  »  Standard  Deviation)  Su  -  Survey)  U  *  University. 
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